akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
On torch.onnx.export()

hitbuyi: <p>I need some help on parameters of  torch.onnx.export(…)</p>
<p>1, dynamic axes.what are backgrounds or application requirements for us to set dynamic axes?</p>

ptrblck: <p><code>dynamic_axes</code> can be used to specify dimensions with a dynamic shape (i.e. the shape is known at runtime and can change). Usually dynamic shapes are used in a temporal dimension or spatial dimensions.</p>

hitbuyi: <p>I’m still not very clear, take temporal as example, if we take time as a dimension of input ,we can write the input tensor as</p>
<p>input_image = [W H T]</p>
<p>where  W is width, H is height, T is temporal</p>
<p>dimension of input_image is 3(fixed), the value of T is changed  at every frame, in this example ,we should set T as dynamic axis?</p>
<p>What I understand about dynamic shape is changing of dimensions, not value of a specified dimension, e.g.,</p>
<p>at frame 1:  input_image = [W H T]<br>
at frame 2:  input_image =[W H]</p>
<p>this is called dynamic shape of input, right?</p>

ptrblck: <aside class="quote no-group" data-username="hitbuyi" data-post="3" data-topic="156882">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/ee59a6/48.png" class="avatar"> hitbuyi:</div>
<blockquote>
<p>What I understand about dynamic shape is changing of dimensions, not value of a specified dimension,</p>
</blockquote>
</aside>
<p>No, usually you refer to a change in the actual shape. I.e. in one iteration the input shape could be <code>[1, 1, 1]</code> in the next it could be <code>[2, 3, 4]</code>.<br>
Dropping or adding an entire dimension would usually just fail at a layer input as a specific number of dimensions is expected.</p>

hitbuyi: <p>Now I get the point, thanks a lot</p>
Mismatch in output size for LSTMs by varying device

Samuel_Horvath: <p>Using the code below produces the correct output shape as described by <a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">LSTM functionality</a> with <code>batch_first=True</code>.</p>
<blockquote>
<p>lstm = nn.LSTM(<br>
input_size=8,<br>
hidden_size=512,<br>
batch_first=True,<br>
num_layers=2)<br>
x = torch.randn(4, 80, 8)<br>
h0 = torch.randn(2, 4, 512)<br>
c0 = torch.randn(2, 4, 512)<br>
lstm_out, hidden = lstm(x, (h0, c0))<br>
print(lstm_out.shape)</p>
</blockquote>
<p>yields</p>
<blockquote>
<p>torch.Size([4, 80, 512]).</p>
</blockquote>
<p>Simply changing the device breaks the functionality by swapping the dimensionality of batch size and sequence length. See below:</p>
<blockquote>
<p>lstm = nn.LSTM(<br>
input_size=8,<br>
hidden_size=512,<br>
batch_first=True,<br>
num_layers=2).to(‘mps’)<br>
x = torch.randn(4, 80, 8).to(‘mps’)<br>
h0 = torch.randn(2, 4, 512).to(‘mps’)<br>
c0 = torch.randn(2, 4, 512).to(‘mps’)<br>
lstm_out, hidden = lstm(x, (h0, c0))<br>
print(lstm_out.shape)</p>
</blockquote>
<p>yields</p>
<blockquote>
<p>torch.Size([80, 4, 512]).</p>
</blockquote>
<p>Is this the known error? I ran the code with PyTorch 1.12 installed through <code>pip</code> on MacBook Pro with Apple M1 Pro chip and MacOS Monterey 12.4.</p>

thecho7: <p>MPS has many issues as it is recently developed.<br>
I don’t think this is the known error, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Seems to be related to <a href="https://discuss.pytorch.org/t/lstm-output-transposed/154820/6">this issue</a>.</p>

PBAL: <p>The nightly build of 1.13 has now fixed this issue. However, note, mps currently does not support LSTM backprop in any case. You can only infer.</p>
How to attach a tensor to a particular point in the computation graph?

Aka: <p>As stated in the question, I need to attach a tensor to a particular point in the computation graph in Pytorch.</p>
<p>What I’m trying to do is this: while geting outputs from all mini-batches, accumulate them in a list and when one epoch finishes, calculate the mean. Then, I need to calculate loss according to the mean, therefore backpropagation must consider all these operations.</p>
<p>I am able to do that when the training data is not much (without detaching and storing). However, this is not possible when it gets bigger. If I don’t detach output tensors each time, I’m running out of GPU memories and if I detach, I lose the track of output tensors from the computation graph. Looks like this is not possible no matter how many GPUs I have since Pytorch does only use first 4 for storing output tensors if I don’t detach before saving them into a list even if I assign more than 4 GPUs.</p>
<p>Any help is really appreciated.</p>
<p>Thanks.</p>

Yuze_Lou: <p>Have you got any solutions on this?</p>
Import torch.nn.functional as F ModuleNotFoundError: No module named 'torch.nn'

mark87: <p>Hi.</p>
<p>Keep getting this error despite installing pytorch as instructed:</p>
<p>—&gt; 10 import torch.nn.functional as F<br>
11 from torch import nn, einsum<br>
12 import torchvision.transforms as T</p>
<p>ModuleNotFoundError: No module named ‘torch.nn’</p>

ptrblck: <p>What does <code>import torch</code> and <code>print(torch.__path__)</code> return? Also, is your <code>wdir</code> set to the PyTorch install directory?<br>
Could you also check if you’ve created local files with conflicting names, such as <code>torch.py</code>?</p>
I got this error while trying to convert nn.module to torchscript.

2681506: <p>terminate called after throwing an instance of ‘torch::jit::ErrorReport’<br>
what():<br>
Unknown builtin op: aten::_convolution_mode.<br>
Could not find any similar ops to aten::_convolution_mode. This op may not exist or may not be currently supported in TorchScript.<br>
:<br>
/Users/yanerrol/opt/anaconda3/envs/py366/lib/python3.6/site-packages/torch/nn/modules/conv.py(298): _conv_forward<br>
/Users/yanerrol/opt/anaconda3/envs/py366/lib/python3.6/site-packages/torch/nn/modules/conv.py(301): forward<br>
/Users/yanerrol/opt/anaconda3/envs/py366/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward<br>
/Users/yanerrol/opt/anaconda3/envs/py366/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl<br>
/Users/yanerrol/Desktop/DGA/cnnlstm_torch/model2model.py(36): forward<br>
/Users/yanerrol/opt/anaconda3/envs/py366/lib/python3.6/site-packages/torch/nn/modules/module.py(1090): _slow_forward<br>
/Users/yanerrol/opt/anaconda3/envs/py366/lib/python3.6/site-packages/torch/nn/modules/module.py(1102): _call_impl<br>
/Users/yanerrol/opt/anaconda3/envs/py366/lib/python3.6/site-packages/torch/jit/_trace.py(965): trace_module<br>
/Users/yanerrol/opt/anaconda3/envs/py366/lib/python3.6/site-packages/torch/jit/_trace.py(767): trace<br>
/Users/yanerrol/Desktop/DGA/cnnlstm_torch/model2model.py(52): <br>
Serialized   File “code/<strong>torch</strong>/torch/nn/modules/conv.py”, line 12<br>
bias = self.bias<br>
weight = self.weight<br>
input = torch._convolution_mode(argument_1, weight, bias, [1], “same”, [1], 1)<br>
~~~~~~~~~~~~~~~~~~~~~~~ &lt;— HERE<br>
return input</p>
<p>Aborted</p>
<hr>
<p>pytorch==1.10.2</p>

ptrblck: <p>Are you seeing the same issue using the latest stable or nightly release?</p>
Type mismatch for layer.bias

shivance: <p>Hi,<br>
I’m working on a CNN implementation in PyTorch, in <a href="https://github.com/deepchem/deepchem/pull/2963" rel="noopener nofollow ugc">this</a> PR. I’m getting mypy error in CI Test.<br>
The mypy check fails saying :</p>
<pre><code class="lang-auto">deepchem/models/torch_models/cnn.py:208: error: Argument 1 to "constant_" has incompatible type "Optional[Tensor]"; expected "Tensor"
Found 1 error in 1 file (checked 364 source files)
Error: Process completed with exit code 1.
</code></pre>
<p>I believe the cause of error is :<br>
In ConvNd layers definition in PyTorch source code <a href="https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#_ConvNd" rel="noopener nofollow ugc">here</a> , <code>bias</code> has type <code>Optional[Tensor]</code> .</p>
<p><code>nn.init.constant_</code> , expects torch.Tensor argument</p>
<p>Is there any way to fix this mypy error ?</p>
MDLSTM Implementation

anto1481: <p>Hello,</p>
<p>I was trying to replicate <a href="https://arxiv.org/abs/1604.08352" rel="noopener nofollow ugc">this model</a>, which uses MDLSTM cells (whose formulation I found <a href="http://www.tbluche.com/scan_attend_read.html" rel="noopener nofollow ugc">here</a>). However, I’m struggling to find any implementation and trying to bring this formulation into a custom layer of the PyTorch framework. I also found a thread in this forum about the topic, but didn’t get much from it.</p>
<p>So, I would like to ask if anyone knows if there is any implementation of the MDLSTM networks in this framework or knows how to implement their formulation.</p>
<p>Thanks in advance.</p>
Skip connections

Berny: <p>Hello. I’m using skip connections in a VAE:</p>
<pre><code>    def _encode(self, x):
          res1e = x
          lin1 = self.relu(self.lin_bn1(self.linear1(x)))
          lin2 = self.relu(self.lin_bn2(self.linear2(lin1)))
          lin2 = lin2 + self.skip(res1e)
</code></pre>
<p>I get this error: The size of tensor a (4096) must match the size of tensor b (10694) at non-singleton dimension 1.<br>
Since VAE considers to reduce the features dimension, is possible to solve this error?</p>

JuanFMontesinos: <p>It’s not an error but a wrong design from your side.<br>
You need to  figure out a system which leads that matches the amount of feats in the output and the input.</p>

Andrei_Cristea: <p>Just to add a keyword to what Juan already said, do a search for “downsampling”, as that’s what you’ll need to implement here. Essentially you’ll need to pick some <code>downsample</code> function that reduces 10694 dimensions down to 4096. Conventionally you could use a <code>AvgPool</code> family layer:</p>
<pre><code class="lang-auto">lin2 = lin2 + downsample(res1e)
</code></pre>

Berny: <p>Good! Thank you very much.</p>
Dataloader generate random pairs

Erica_Zheng: <p>Dear All,</p>
<p>I am working on domain adaptation and looking for a solution to generate random pairs.</p>
<p>Xs = [xs_1, xs_2, …, xs_m]<br>
Xt = [xt_1, xt_2, …, xt_n]</p>
<p>I want to Dataloader to output random pairs e.g.</p>
<p>(xs_2, xt_3), (xs_1, xt_6), …</p>
<p>Thank you.</p>

JuanFMontesinos: <p>Create a custom dataloader class and use any random package (python has a default one) to generate and integer to index x_s and x_t</p>
<p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class</a></p>
Dataloaders does not "start"

thistlillo: <p>Please consider this simple fragment of code:</p>
<pre><code class="lang-auto">[...]
print("resetting metrics")
for m in self.metrics:
      m.reset()
 print("done")

 for bi, (images, labels, text) in enumerate(dl):
       if bi == 0:
           print("IMAGES:", images.shape)
           print("LABELS:", labels.shape)
           print("TEXT:", text.shape)
[...]
</code></pre>
<p>Normally, my code runs fine, but, every once in a while, it gets stuck.  Indeed, the only output is:</p>
<pre><code class="lang-auto">* EPOCH 1/1000, START
calling concrete _epoch() method
resetting metrics
done
</code></pre>
<p>and nothing else. If I interrupt the process with <code>CTRL+C</code>, I read this:</p>
<pre><code class="lang-auto">[...]
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/opt/conda/envs/torch/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
</code></pre>
<p>that seems to indicate it was stuck in <code>popen_fork.py</code> waiting for some other process to finish.</p>
<p>I know the fragment is very small, but is there anyone who knows what I could do to try and solve this problem? I cannot reproduce it, meaning that it normally runs fine, but sometimes it does not.</p>
<p>I do not use explicitly any multiprocessing functions, just “plain” pytorch code. However, the <code>DataLoader</code> is configured to use multiple workers.</p>
Adjusting Keypoint R-CNN or Finetuning it end-to-end with a GCN

ATAboukhadra: <p>Hi,</p>
<p>I’m using <code>KEYPOINTRCNN_RESNET50_FPN</code> from the torchvision library to estimate pose from RGB images. The model is successful but it lacks the ability to correlate keypoints together so it causes many flying points. One solution is to use graph-like networks to find relations between those keypoints. The problem is that the Keypoint RCNN in training mode is a standalone model trained separately from any other component and has its own loss function and it only produces losses. How can I add a graph network and train it end-to-end or adjust a layer within the model to be a graph network?</p>
<p>Do you have any advice on which layer I should adjust and how, or whether it’s possible to fine-tune this model with another one?</p>
<p>Thanks a lot.</p>

ATAboukhadra: <p>My solution to this was by making copies of the source code of the Keypoint RCNN and ROI heads and other relevant files from the torchvision library to adjust the model to my problem.</p>
Mask RCNN number of predictions

ATAboukhadra: <p>Hi,</p>
<p>I’m using the Keypoint RCNN from torchvision to estimate bounding boxes and keypoints for objects in RGB images. In my dataset, it’s guaranteed that there is 1 and only 1 object in the image. Therefore, I’m not interested in the ability of the RCNN to produce all possible relevant RoIs, I’m only interested in only 1.</p>
<p>To do so, I set rpn_post_nms_top_n_train to 1. <strong>This works fine but sometimes the RPN produces 2 RoIs per image during training even though I specified that I only need 1. How do I fix that? What is this<br>
second RoI?</strong></p>
<p>My second question is related to the same problem. With more training, the RCNN shows a pattern of overfitting where it stops giving predictions to avoid more loss so the number of RoIs becomes 0 for some testing images.<br>
It’s guaranteed that every sample in the test set has an object. <strong>How do I penalize the RCNN if it doesn’t produce any outputs or how can I guarantee that it gives exactly 1 prediction per image (no more no less)?</strong></p>
<p>Another related question:<br>
Which parts of the RCNN can be freezed to maintain the number of predictions per image while finetuning the rest of it?</p>
<p>Thanks a lot.</p>
How to delete all the gradients in between operations

malfonsoarquimea: <p>Hi! I am currently working on a project where, for a given trained model, I perform inference batches of inputs and compute (and store) the gradients of the output with respect to the inputs.<br>
My code is something like this</p>
<pre><code class="lang-auto">gradients_list=[]
for batch_of_inputs in batches:
   batch_of_inputs.requires_grad_()
   output=model(batch_of_inputs)
   gradients = torch.autograd.grad(
            outputs=output,
            inputs=batch_of_inputs,
            grad_outputs=torch.ones_like(output),
            retain_graph=False,
        )
   gradients_list.append(gradients.detach_())
   # here I need to do something to delete all the gradients
</code></pre>
<p>The thing is that the used memory increases and increases until OOM error rises. I have tried to use</p>
<pre><code class="lang-auto">del gradients,batch_of_inputs,output
</code></pre>
<p>and the problem persisted.<br>
What would you suggest?<br>
Thanks in advance</p>

ptrblck: <p>How large is the increase in memory in each iteration?<br>
Since you are storing the gradients in each iteration, an increase is of course expected which would depend on the size of the gradient tensor. Could you check its size and compare it to the actual increase?</p>

malfonsoarquimea: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> the tensor I am storing is a 3x100 flotat 32 tensor, which should be 300/8*32=1.2KB, while the memory increase is 20-60 MB per step (it varies, don’t know why).</p>
Torch and list data as learning features

lanka: <p>Hi,<br>
i need some explanation about list data ,<br>
here I am using transfer learning features(fc) suppose 1024 features(saved into pickle file). i have saved all features and later imported them into <code>fearure=[]</code> (as list data)</p>
<pre><code class="lang-auto">for img in features:
    img=torch.FloatTensor(img)
    output=model(img)
</code></pre>
<p>do I need set <code>img.requires_grad=True</code>, before feed to the model?</p>

ptrblck: <p>Usually you wouldn’t need to set <code>requires_grad=True</code> on the inputs unless you explicitly want to update the inputs for e.g. an adversarial use case.</p>

lanka: <p>Ok. Thanks. I just need to put them in as “the thought vector” in LSTM.</p>
Error: could not convert ‘c10::nullopt’ from ‘const c10::nullopt_t’ to ‘c10::optional<c10::MemoryFormat>’ Error occurs in Make period

Fuzhiyuan: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/ca731d2b9d37dddcc66f091b976beb97b301e914.png" data-download-href="https://discuss.pytorch.org/uploads/default/ca731d2b9d37dddcc66f091b976beb97b301e914" title="Screenshot from 2022-04-29 14-52-49"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca731d2b9d37dddcc66f091b976beb97b301e914_2_690x393.png" alt="Screenshot from 2022-04-29 14-52-49" data-base62-sha1="sSX4VGPRwJAPtxy9gpUiIpElkJC" width="690" height="393" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca731d2b9d37dddcc66f091b976beb97b301e914_2_690x393.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca731d2b9d37dddcc66f091b976beb97b301e914_2_1035x589.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/ca731d2b9d37dddcc66f091b976beb97b301e914_2_1380x786.png 2x" data-dominant-color="3B172E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-04-29 14-52-49</span><span class="informations">1848×1053 203 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>version:<br>
cuda:           11.3<br>
cudnn:          cudnn-11.3-linux-x64-v8.2.1.32<br>
libtorch：     libtorch-shared-with-deps-1.11.0+cu113<br>
python：     3.8</p>
<p>The following is cmakelist:</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0.0 FATAL_ERROR)
project(libtorch_test)
message(STATUS "${CMAKE_CUDA_ARCHITECTURES}")
set(CMAKE_CUDA_ARCHITECTURES 70 75 80)
set(CUDNN_LIBRARY_PATH "/usr/local/cuda-11.3/lib64")
set(CUDNN_INCLUDE_PATH "/usr/local/cuda-11.3/include")
message(STATUS "SET SUCESSFUL!")
message(STATUS "${CMAKE_CUDA_ARCHITECTURES}")
set(CMAKE_PREFIX_PATH "/home/liangxijie/Lib/libtorch-shared-with-deps-1.11.0+cu113/libtorch/share/cmake/Torch")
find_package(Torch REQUIRED)
message(STATUS "Pytorch status:")
message(STATUS "libraries: ${TORCH_LIBRARIES}")
add_executable(libtorch_test deploy.cpp)
target_link_libraries(libtorch_test "${TORCH_LIBRARIES}")
set_property(TARGET libtorch_test PROPERTY CXX_STANDARD 11)

</code></pre>

zww_0108: <p>have you got it?I miss the same problem,hope your help</p>

Fuzhiyuan: <p><a class="mention" href="/u/zww_0108">@zww_0108</a>   Sorry, I solved this problem by using vs, and I just set some configurations in vs. Thus, I think you could look through the procedure of installing libtorch in vs on win and find out how these configurations are set on CMake.</p>
F.pad() backward pass is too slow

NguyenHai7120: <p>Hello everyone,</p>
<p>I’m trying to define a function that implements an operator involving a convolution.So I used the functional function F.conv2d() to implement my operator. By default, I used the <em>zeros_padding</em> mode and there is no problem. But when I switched into <em>reflect</em> or <em>circular</em> padding mode using the function F.pad(), the backward pass was too slow.</p>
<p>Does anyone have a solution with this F.pad() backward pass?<br>
Thanks in advance!</p>

soulitzer: <p>How slow are we talking here? Could you post benchmarking results?</p>
Conversion of 2-D metric to 1-D vector

ArshadIram: <p>I have a 2-D matrix of activation values for every image. I want to made up 1-D activation vectors corresponding to each image. I am not sure how to do it. Any help regarding this would be great.</p>

thecho7: <p>Is this topic related to distributed?<br>
Anyway, could you give us more details about the activation value?</p>
<p>Describing it with a dimension would be helpful.</p>

ArshadIram: <p>I updated the category. Thanks for indicating.</p>
<p>The activation values are the logits. The dimension is 8749 rows × 11 columns.</p>

thecho7: <p>There are several ways to make the tensor flatten.<br>
Among them, in this case,</p>
<ol>
<li>
<code>nn.Flatten</code> would be the one you are looking for.<br>
(<a href="https://pytorch.org/docs/stable/generated/torch.nn.Flatten.html#torch.nn.Flatten" class="inline-onebox" rel="noopener nofollow ugc">Flatten — PyTorch 1.12 documentation</a>)</li>
<li>
<code>tensor.view()</code> is the popular option. <code>activation_vector.view(batch_size, -1)</code> makes the tensor flatten.</li>
</ol>
<p>For any case, keep in mind that your tensor should have at least 2-D which indicates the size of batch in the first dimension.</p>
How to multiply a batch of lists of vectors by a list of matrices?

arachnidman: <p>I have a batch of lists of vectors<br>
[[v_11, v_12, …, v_1n], [v_21, v_22, …, v_2n], …]<br>
and a list of matrices<br>
[M_1, M_2, …, M_n]<br>
and I want to compute the product<br>
[[M_1 v_11, M_2 v_12, …, M_n v_1n], [M_1 v_21, M_2 v_22, …, M_n v_2n], …].</p>
<p>Is there a vectorized way to do this?</p>

AlphaBetaGamma96: <p>You could try using <code>torch.stack</code> to convert your list of vectors and list of matrices into 2 different Tensors. Then just apply <code>@</code> or <code>torch.einsum</code> to perform the matrix multiplication.</p>
<pre><code class="lang-auto">import torch

list1 = [torch.randn(5) for _ in range(10)]
list2 = [torch.randn(5) for _ in range(10)]

matrix1 = [torch.randn(5,5) for _ in range(10)]

list1tensor = torch.stack(list1, dim=0)
list2tensor = torch.stack(list2, dim=0)
matrix1tensor = torch.stack(matrix1, dim=0)

mv1 = torch.einsum("bij,bj-&gt;bi",matrix1tensor, list1tensor)
mv2 = torch.einsum("bij,bj-&gt;bi",matrix1tensor, list2tensor)

</code></pre>
Modifying the source code of Pytorch

Vishesh_Rangwani: <p>I want to find in which C file the thread is created when I use <code>torch.set_num_thread()</code> . For this I did a grep in pytorch source code and then in all those places where there was an occurrence of set_num_threads, I added a <code>printf()</code> . I built it again and still cannot have anything on the terminal from the <code>printf()</code> statements added.</p>
<p>I got a suggestion to debug my code using gdb, but I am unaware of how to use the gdb for source code.</p>
<p>It would be great if someone could suggest how to use the debugger and also if someone knows why arent the <code>printf()</code> working.</p>

ptrblck: <p>After adding the <code>printf</code> statements to the C++ backend, I assume you’ve successfully rebuilt PyTorch from source? If not, these changes won’t be visible as they need to be compiled first.</p>
<p><a href="https://www.cs.cmu.edu/~gilpin/tutorial/">This small <code>gdb</code> tutorial</a> might be a good a good starter.</p>

Vishesh_Rangwani: <p>Yes I did rebuilt using <code>sudo python3 setup.py develop</code></p>

ptrblck: <p>In that case the methods might not be called at all. Add another print statement to a call which you know will be executed and make sure you are really running your source build first.</p>

Vishesh_Rangwani: <p>I am really not sure about any of the methods which will definitely be called. I will attach the code which I am executing (a simpler version of that).</p>
<p>I also want to point out the fact that, I had done the same changes of adding <code>printf()</code> to certain files in my Mac and rebuilt it. At that time, the changes were being reflected and I could see the printf() executing.</p>
<p>Here is a simple code which does show the effect of adding <code>printf()</code> on mac but not on linux.</p>
<pre><code class="lang-auto">import timeit
import torch
import traceback

t = 4


torch.set_num_threads(t)

r = timeit.timeit(setup = "import torch; x = torch.randn(259, 259); y = torch.randn(259, 259)", stmt="torch.mm(x, y)", number=100)
runtimes.append(r)
print(t, r)


print(torch.__config__.parallel_info())
</code></pre>
Why does `nn` has `ReLU`, `Tanh` and 'Sigmoid` non-linear layers but not `Atan`?

Fei_Liu: <p>I mean these are all non-linear transformations and they can all be handily accessed with <code>Tensor.xxx</code> (for <code>ReLU</code> it’s <code>Tensor.clip(min=0.0)</code> so only a minor difference). It looks to me they share many similarities. Hence the question: are there any special reasons that <code>atan</code> only exists in utility and tensor member functions but not an <code>nn.Module</code> like <code>nn.Tanh</code> ?</p>

thecho7: <p>I don’t think there is any special reason but atan is not that famous…</p>
<p>Is there any advantages? Converge to pi / 2 is not effective in training</p>

Fei_Liu: <p>I think atan is “slightly” better than tanh in some cases since it’s derivative decays slower (1/x^2 decay v.s. exp decay) so it suffers less from the dead neuron problem at least on paper. In practice it all depends on the specific problems of course. I don’t think there’s issue in converging to pi/2 or 1 or some other constant as it can be easily rescaled.</p>

thecho7: <p>Oh, I see.<br>
But PyTorch doesn’t support atan yet.</p>

ptrblck: <p>PyTorch does support <code>atan</code>.</p>
<p><a class="mention" href="/u/fei_liu">@Fei_Liu</a><br>
You can easily define a custom module via:</p>
<pre><code class="lang-python">class Atan(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return torch.atan(x)

model = nn.Sequential(
    nn.Linear(10, 10),
    Atan(),
    nn.Linear(10, 2)
)

x = torch.randn(1, 10)
out = model(x)
out.mean().backward()
print([(name, p.grad.abs().sum()) for name, p in model.named_parameters()])
</code></pre>
<p>If you think this is a widely used non-linearity and deserves an own implementation, feel free to create a feature request on GitHub.</p>

thecho7: <p>Thanks to your reply.<br>
What I meant is there is no <code>nn.Atan()</code> yet <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Fei_Liu: <p>Thanks. That I understand, and it’s exactly what I’m doing now in my code. This question is purely out of curiosity on why it’s not originally supported in the <code>nn</code> library as the other common non-linear activators. I do agree that <code>atan</code> is somewhat less common than <code>tanh</code> in the context of activation functions. Maybe that’s just the reason.</p>

ptrblck: <p>Yes, I think you are right and it might not be common enough (so far).<br>
If you create the feature request, make sure to add some papers or popular repositories using <code>atan</code> as an activation function.<br>
We had a similar use case for <code>nn.Identity</code> which is trivial to implement, but was commonly used so that a built-in module made sense.</p>
RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 4 but got size 3 for tensor number 1 in the list

Adwait_Naik: <p>Hi ya,</p>
<p>I got this error while training my NeRF on llff synthetic image dataset.</p>
<p>Error:</p>
<pre><code class="lang-auto">RuntimeError: Sizes of tensors must match except in dimension 0. Expected size 4 but got size 3 for tensor number 1 in the list.
</code></pre>
<p>Code:</p>
<pre><code class="lang-auto">dummy_num = ((len(poses) - 1) // args.num_gpu + 1) * args.num_gpu - len(poses)
dummy_poses = torch.eye(3, 4).unsqueeze(0).expand(dummy_num, 3, 4).type_as(render_poses)
print(f"Append {dummy_num} # of poses to fill all the GPUs")
with torch.no_grad():
  nerf.eval()
   rgbs,_ = nerf(H, W, K, args.chunk, poses=torch.cat([poses, dummy_poses], dim=0).cuda(),
                               render_kwargs=render_kwargs_test)
   rgbs = rgbs[:len(rgbs) - dummy_num]
   rgbs_save = rgbs  # (rgbs - rgbs.min()) / (rgbs.max() - rgbs.min())
</code></pre>
<p>The error is in this line</p>
<pre><code class="lang-auto">rgbs,_ = nerf(H, W, K, args.chunk, poses=torch.cat([poses, dummy_poses], dim=0).cuda(),
                               render_kwargs=render_kwargs_test)
</code></pre>
<p>Could you help, <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Why does LBFGS scale the initial step size?

benjww: <p>I’m currently employing the LBFGS learning algorithm without line search and noticed the initial step size may be scaled relative to the gradient:</p>
<pre><code class="lang-auto">############################################################
# compute step length
############################################################
# reset initial guess for step size
if state['n_iter'] == 1:
    t = min(1., 1. / flat_grad.abs().sum()) * lr
else:
    t = lr
</code></pre>
<p>Does anyone know the reasoning for this? Most implementations I’ve seen (that don’t use line search) employ a fixed step size. I’ve also looked in the appropriate chapters in [Jorge Nocedal, Stephen Wright: Numerical Optimization] and found no discussion of scaling step size in this way.</p>
<p>Thanks!</p>
Bfloat16 training question

NikitaBezukhov: <p>Yo, guys, I am looking to train a 3D CNN on my RTX3080ti and as I know it supports bfloat16, but I struggle to train a model. I tried casting everything straight to bfloat 16 and I tried using bfloat16 with AMP but it throws an error, it sounds like “ illegal memory access” and when I am just using fp32 everything is fine. Any tips on bfloat16 training and is it even viable? I have seen that pytorch lightning has some king of support, but I would prefer normal torch.</p>

ptrblck: <p>Could you post a minimal, executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

NikitaBezukhov: <p>Yes, I will try to do it in a day or two)</p>

Tin: <p>Hey, did you solve this issue?</p>

NikitaBezukhov: <p>Nah, Im am just using normal AMP training with float16</p>
Jit.script cause bad graph conversion from Torch to Onnx

Davide_Orru: <p>hello, I wrote this Torch script to populate a correlation matrix:</p>
<p><span class="mention">@torch.jit.script</span><br>
def populate_corr(corr: torch.Tensor,<br>
input1: torch.Tensor,<br>
input2: torch.Tensor,<br>
max_displacement: Tuple[int, int],<br>
stride: Tuple[int, int],<br>
dilation_patch: Tuple[int, int],<br>
h: int,<br>
w: int):</p>
<pre><code>for i in range(0, 2*max_displacement[0]+1, dilation_patch[0]):
    for j in range(0, 2*max_displacement[1]+1, dilation_patch[1]):
        p2 = input2[:, :, i:i+h, j:j+w]
        p2 = p2[:, :, ::stride[0], ::stride[1]]
        corr[:, i//dilation_patch[0], j//dilation_patch[1]] = (input1 * p2).sum(dim=1)
return corr
</code></pre>
<p>When I export the torch model to onnx model size is kept and also inference is the same but exploring the onnx model in netron shows that the resulting graph inside the converted the onnx model is very convoluted and full of complex subgraphs caused by such loops, this is translated in poor speed performances once the model is deployed on end device (Torch → onnx → TF → tflite quantized).</p>
<p>Is there a more efficient way to write such script in order to obtain a more optimized graph?</p>
Model size after pruning

54WC: <p>The model is pruned after training. I suppose the pruned model should has less size than original model, right?  But it actually become double after pruning.  Is there a right way can really reduce model size?</p>
<p>mymodel.pth  72.8MB<br>
my model_pruned.pth 136.1MB</p>
<pre><code class="lang-auto">def get_model():
    model = models.vgg16(pretrained=True)
    for param in model.parameters():
        param.requires_grad = False
    model.avgpool = nn.Sequential( nn.Conv2d(512,512,3),
      nn.MaxPool2d(2),
      nn.Flatten())
    model.classifier = nn.Sequential(
      nn.Linear(2048, 512),
      nn.ReLU(),
      nn.Dropout(0.5),
      nn.Linear(512, 136),
      nn.Sigmoid()
    )
    
    criterion = nn.L1Loss()
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
    return model.to(device), criterion, optimizer

def prun_(model):
    parameters_to_prune = (
       (model.features[0],'weight'),
       (model.features[2],'weight'),
       (model.features[5],'weight'),
       (model.features[7],'weight'),
       (model.features[10],'weight'),
       (model.features[12],'weight'),
       (model.features[14],'weight'),
       (model.features[17],'weight'),
       (model.features[19],'weight'),
       (model.features[21],'weight'),
       (model.features[24],'weight'),
       (model.features[26],'weight'),
       (model.features[28],'weight'),
       (model.classifier[0],'weight'),
       (model.classifier[3],'weight'),
    )
    prune.global_unstructured(
        parameters_to_prune,
        pruning_method=prune.L1Unstructured,
        amount=0.4,
    )

model.eval()
print(model.state_dict().keys())
traced_script_module.save("mymodel.pt")

prun_(model)
print(model.state_dict().keys())
torch.save(model.to('cpu').state_dict(), 'mymodel_pruned.pth')

</code></pre>

54WC: <p>I find the pruned model include additional “original” and “mask” data. It could be the reason that file become larger. Could I throw away these data?  Is there a convenient to do that?  Thanks!</p>

Lezcano: <p>If you want to remove the pruning afterwards you can do <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.prune.remove.html#torch.nn.utils.prune.remove" rel="noopener nofollow ugc"><code>torch.nn.utils.prune.remove</code></a></p>

54WC: <p>Finally,I can use [`torch.nn.utils.prune.remove’] to remove the pruning.<br>
The original and pruned model have the same size.  I test one forward pass, the original model takes 0.0995msec, and the pruned model takes 0.102msec.<br>
it trun out the model size are the same, but implement time are almost the same. Why pruning? It is quite strange.</p>
Can`t obtain right data from tensor.contiguous().data_ptr()

Fuzhiyuan: <p>I want to print the data from torch::Tensor.contiguous() using deference, but it returns a wrong result which is 1.4013e-43. This tensor shape is 663x663x860 in CPU. <strong>If I create a new tensor using torch::rand({600,600,600}) to run this code result would be correct.</strong></p>
<p>This is my project code:</p>
<pre><code class="lang-auto">std::cout &lt;&lt; "original address: " &lt;&lt; static_cast&lt;float*&gt;(data.data_ptr()) &lt;&lt; std::endl;

torch::Tensor contiguous_tensor = data.contiguous();

auto contiguous_address = static_cast&lt;float*&gt;(contiguous_tensor.data_ptr());

std::cout &lt;&lt; "contiguous address: " &lt;&lt; static_cast&lt;float*&gt;(contiguous_tensor.data_ptr()) &lt;&lt; std::endl;

contiguous_tensor[0][0][2] = 100;

torch::Tensor nz = torch::nonzero(data);
long long  a = nz[0][0].item&lt;int&gt;();
int b = nz[0][1].item&lt;int&gt;();
int c = nz[0][2].item&lt;int&gt;();

std::cout &lt;&lt; "a b c value is :" &lt;&lt; data[a][b][c] &lt;&lt; std::endl;
std::cout &lt;&lt; "contiguous tensor nonzero is : " &lt;&lt; contiguous_tensor[a][b][c] &lt;&lt; std::endl;
std::cout &lt;&lt; "contiguous tensor pointer value is : " &lt;&lt; *(contiguous_address + a * size[2] * size[1] + b * size[2] + c) &lt;&lt; std::endl;
std::cout &lt;&lt; "prank value is : " &lt;&lt; *(contiguous_address + 2) &lt;&lt; std::endl;

for (int i = 0; i &lt; 20; i++)
	std::cout &lt;&lt; "address + " &lt;&lt; i &lt;&lt; " is : " &lt;&lt; *(contiguous_address + i) &lt;&lt; std::endl;
</code></pre>
<p>And it`s result:</p>
<pre><code class="lang-auto">original address : 000001C25E3E6080
contiguous address : 000001C31281E080
a b c value is :5
[ CPULongType{} ]
contiguous tensor nonzero is : 5
[ CPULongType{} ]
contiguous tensor pointer value is : 0
prank value is : 0
address + 0 is : 0
address + 1 is : 0
address + 2 is : 0
address + 3 is : 0
address + 4 is : 1.4013e-43
address + 5 is : 0
address + 6 is : 0
address + 7 is : 0
address + 8 is : 0
address + 9 is : 0
address + 10 is : 0
address + 11 is : 0
address + 12 is : 0
address + 13 is : 0
address + 14 is : 0
address + 15 is : 0
address + 16 is : 0
address + 17 is : 0
address + 18 is : 0
address + 19 is : 0
</code></pre>

Fuzhiyuan: <p>The most confusing is i have set contiguous_tensor = data.contiguous().to(torch::long), but the output is still float number, 1.403e-45. Theoretically, a tensor with long DataType should output 0 at this case instead of 1.403e-45.</p>
Get device id of DLA

sn710: <p>When running Pytorch inference on a Resnet model on Jetson Xavier GPU, in my python script I use -</p>
<pre><code class="lang-auto">device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
</code></pre>
<p>so that I can later do something like - <code>inputs = inputs.to(device)</code> to copy the inputs from host to device (cpu to gpu) before running the inference.</p>
<p>Similarly, when I want to use the 2 DLAs on the Jetson for my inference, how should I find the device id?</p>
<p>When i try -</p>
<pre><code class="lang-auto">print ('Available devices ', torch.cuda.device_count())
print ('Current cuda device ', torch.cuda.current_device())
print('GPU Device name:', torch.cuda.get_device_name(torch.cuda.current_device()))
</code></pre>
<p>I get</p>
<pre><code class="lang-auto">Available devices  1
Current cuda device  0
GPU Device name: Xavier
</code></pre>
<p>Trying something like</p>
<pre><code class="lang-auto">device = torch.device('dla:0' if torch.cuda.is_available() else 'cpu')
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">RuntimeError: Expected one of cpu, cuda, xpu, mkldnn, opengl, opencl, ideep, hip, ve, ort, mlc, xla, lazy, vulkan, meta, hpu device type at start of device string: dla
</code></pre>
<p>I can verify that the DLA is present, as I can run tensorrt inference on it. Could somebody please guide me on how to do it via pytorch?</p>

ptrblck: <p>The <code>DLA</code> is not a built-in device in PyTorch and you could use it e.g. via TensorRT.</p>

sn710: <p>So, via only Pytorch, there is no way to access the DLA.<br>
So far, tensorrt seems to be the only framework through which i could access the DLA. Do you know of any other frameworks i could use? Als, is there any standard model you know that wil run on DLA directly without gpu fallback?</p>
How can torchscript C++ parallelize modulelist calculation?

triloon: <p>Hi all,</p>
<p>I want to know how can I parallelize my modulelist in torchscript C++ codes.</p>
<p>I have read that the modulelist is completely unrolled from this blog: <a href="https://pytorch.org/docs/master/jit_language_reference_v2.html" rel="noopener nofollow ugc">https://pytorch.org/docs/master/jit_language_reference_v2.html </a></p>
<ul>
<li>Code that iterates over <code>torch.nn.ModuleList</code> or <code>torch.nn.ModuleDict</code> is completely unrolled so that elements of <code>torch.nn.ModuleList</code> or keys of <code>torch.nn.ModuleDict</code> can be of different subclasses of <code>torch.nn.Module</code>.</li>
</ul>
<p>Does it mean we don’t need to modify the C++ code to enabel parallelism of Modulelist<br>
because the modulelist is unrolled automatically?</p>
<p>On the other hand, I also read from this blog <a href="https://pytorch.org/tutorials/advanced/torch-script-parallelism.html" class="inline-onebox" rel="noopener nofollow ugc">Dynamic Parallelism in TorchScript — PyTorch Tutorials 1.12.0+cu102 documentation</a><br>
that we can use torch.jit.fork and torch.jit.wait to manually parallelize the modulelist calculations. I want to know will the fork &amp;wait also affects C++ torchscript code so that modulelist can be calculated in parallel automatically ?</p>
<p>Finally, if above two methods cannot automatically enable C++ parallel calculation of modulelist, could we implemented a new torchscript custom class that accept some torch.nn.Modules as parameters and parallel these modules manually by ourselves?</p>
<p>Thanks very much for any advices.</p>
When does 'inf' exactly show on?

tolry418: <p>Hello.<br>
I found some different result of calculation between torch and raw In python</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a=torch.tensor(888)
&gt;&gt;&gt; torch.exp(a/10)
tensor(inf)

&gt;&gt;&gt; b=888
&gt;&gt;&gt; math.exp(b/10)
3.6757840844711625e+38
</code></pre>
<p>I thought it is because of overflow of criterion in pytorch.<br>
Why does it show different between them??<br>
When we exactly see that overflow in pytorch?<br>
In pytorch, Even if we allocate tensor in cuda, we also had same result.<br>
Thanks in advance.</p>

thecho7: <p>I think what you actually did is</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.Tensor([888]) # torch.Tensor(888) create a tensor has 888 elements
&gt;&gt;&gt; torch.exp(a / 10)
tensor([inf])
</code></pre>
<p>The difference between two cases are came from the type of data.<br>
PyTorch creates 32bit float tensor by calling <code>torch.tensor</code></p>
<p>Meanwhile, if you install the 64-bit version of Python, Python handles 64-bit float data.</p>
<p><code>exp(88.8)</code> exceeds the upper limit of 32-bit expression but does not exceeds 64-bit expression.<br>
If you want handle that number in PyTorch, use <code>DoubleTensor</code> instead.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.DoubleTensor([888])
&gt;&gt;&gt; torch.exp(a/10)
tensor([3.6758e+38], dtype=torch.float64)
</code></pre>
<p>See docs <a href="https://pytorch.org/docs/stable/tensors.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor — PyTorch 1.12 documentation</a></p>

tolry418: <p><a class="mention" href="/u/thecho7">@thecho7</a><br>
Aha! Thank you for your kind explanation.<br>
What if we want to speed up calculation using 32bit than 64bit.<br>
I think we may get into speed-exact dilemma. what do you think which one is better?</p>

thecho7: <p>Many of recent works using 16-bit tensor but not 64-bit.<br>
Using 64-bit is not a good option in the respect of speed and accuracy.</p>
<p>I recommend you to use 32-bit.</p>

tolry418: <p><a class="mention" href="/u/thecho7">@thecho7</a><br>
Really thanks!!</p>
Change number of channel

sampa: <p>Hi, I want to change number of channels from torch.Size([1, 3, 128, 128])  to torch.Size([1, 1, 128, 128])…How to do that?</p>

thecho7: <p>I guess there are several ways to do that but Separable Convolution is a popular option.</p>
<pre><code class="lang-auto">a = torch.rand(1, 3, 128, 128)
layer = torch.nn.Conv2d(in_channels=3, out_channels=1, kernel_size=1)
out = layer(a)
</code></pre>
Use slice of a tensor as parameters

kzinovjev: <p>Hi. I am trying to define a module that accepts a tensor of parameters but allowing to control the slice of this tensor that will actually be learned, for example:</p>
<pre><code class="lang-auto">class MyModule(nn.Module):
    
    def __init__(self, m, learn_slice):
        super().__init__()
        self.m = m
        self.m_params = nn.Parameter(m[learn_slice])
        self.m[learn_slice] = self.m_params
        
    def forward(self, x):
        return torch.dot(self.m, x)
</code></pre>
<p>When I evaluate the gradient for the first time it works well, but after the second <code>backward()</code> call I get the following error:</p>
<pre><code class="lang-auto">m = torch.rand(100)
x = torch.rand(100)
model = MyModule(m, slice(10, 20))
y = model(x)
y.backward()
y = model(x)
y.backward()
</code></pre>
<blockquote>
<p>RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.</p>
</blockquote>
<p>If I always add <code>retain_graph=True</code> to <code>backward()</code> it does work, but I am not sure whether it is the right way of doing this.</p>
<p>What am I missing here? Thanks!</p>

DBraun: <p>I think this is the way</p>
<pre><code class="lang-python">class MyModule(nn.Module):
    
    def __init__(self, m, learn_slice):
        super().__init__()
        self.register_parameter("m", m)
        self.register_buffer("learn_slice", learn_slice)
        
    def forward(self, x):
        return torch.dot(self.m, x)

    def parameters(self):
        # override nn.Module's parameters
        return self.m[self.learn_slice]
</code></pre>
<p>I think that all of <code>m</code> will be saved with the model, but only the learn_slice section will be affected by optimizers and backprop. See <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.parameters" class="inline-onebox" rel="noopener nofollow ugc">Module — PyTorch 1.12 documentation</a></p>
How could I test torchvision model accuracy?

coincheung: <p>Hi,</p>
<p>I noticed torchvison added swin transformer, and the acc is higher than the experiment result reported in paper:<br>
<a href="https://github.com/pytorch/vision/blob/3db30442a52e250c569ba131485fc432c8300f16/torchvision/models/swin_transformer.py#L485" class="inline-onebox" rel="noopener nofollow ugc">vision/swin_transformer.py at 3db30442a52e250c569ba131485fc432c8300f16 · pytorch/vision · GitHub</a><br>
Would you tell me how could I test the pretrained model downloaded from the official website and achieved that result?</p>
Why tensor cannot be trained initialized all zeros?

Ximeng: <p>Hi!</p>
<p>The sudo simulate network looks like:<br>
inut:x<br>
x = x*att<br>
y = mlp(x)</p>
<p>as for att:<br>
att is initialized with all ones, all rand, or all zeros<br>
att = mlp(relu(att))<br>
att = softmax(att)</p>
<p>When I initialized with torch.ones or torch.rand, it can be trained properly. However if I initialize it with all zeros, it remains the same til the end. Just curious, what is the difference between all ones and all zeros as they are exactly the same after softmax operation at very beginning( say [0.25, 0.25, 0.25, 0.25]).</p>
<p>Thank you！</p>

thecho7: <p>If there is no bias, 0 always makes 0.<br>
Think about how the forward/backward propagations work.</p>

Ximeng: <p>I get the point. Thanks very much.</p>

thecho7: <p>Thanks. Please check the reply as a solution to make this issue solved</p>
Extracting Features in vision models

Tejan_Mehndiratta: <p>Hi,<br>
I am running this piece of code to extract features from an intermediate layer and it’s showing the described error.<br>
Can somebody please help me?</p>
<pre><code class="lang-auto">import torchvision
from torchvision.models.feature_extraction import create_feature_extractor
model_faster_rcnn = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model_faster_rcnn.eval()

faster_rcnn = create_feature_extractor(model_faster_rcnn, {'layer4': 'f1'})
</code></pre>
<p>The error given is:</p>
<pre><code class="lang-auto">TraceError: Proxy object cannot be iterated. This can be attempted when the Proxy 
is used in a loop or as a *args or **kwargs function argument. See the torch.fx docs 
on pytorch.org for a more detailed explanation of what types of control flow can 
be traced, and check out the Proxy docstring for help troubleshooting Proxy 
iteration errors.
</code></pre>
<p>I also tried the feature extraction code for resnet and it worked, but for this, it’s not working.</p>

MartinZhang: <p>Have you solved it?I also met the same problem <img src="https://discuss.pytorch.org/images/emoji/apple/face_with_head_bandage.png?v=12" title=":face_with_head_bandage:" class="emoji" alt=":face_with_head_bandage:" loading="lazy" width="20" height="20"></p>

Tejan_Mehndiratta: <p>You can use Pytorch hooks. Check it out.</p>