akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
TypeError 'Tensor' object is not callable

atharvahans1: <p>Hello,</p>
<p>I am new to PyTorch and I was trying to use the autograd package. I keep getting <code>TypeError: 'Tensor' object is not callable</code>. I am not sure what I am doing wrong. I have attached a screenshot of the error and my code below. Can someone please help?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/8/e82404fa0e85b97ce168e043080f4a322a9e4be6.png" data-download-href="https://discuss.pytorch.org/uploads/default/e82404fa0e85b97ce168e043080f4a322a9e4be6" title="Screen Shot 2020-11-27 at 2.13.34 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e82404fa0e85b97ce168e043080f4a322a9e4be6_2_514x374.png" alt="Screen Shot 2020-11-27 at 2.13.34 PM" data-base62-sha1="x7BWyqpZ299RAy6Swr5yVXAEknA" width="514" height="374" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e82404fa0e85b97ce168e043080f4a322a9e4be6_2_514x374.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e82404fa0e85b97ce168e043080f4a322a9e4be6_2_771x561.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/8/e82404fa0e85b97ce168e043080f4a322a9e4be6_2_1028x748.png 2x" data-dominant-color="F4F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-11-27 at 2.13.34 PM</span><span class="informations">2112×1538 506 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">def lagrangian(q, q_dot, m1, m2, l1, l2, g):
    
    t1, t2 = q     # theta 1 and theta 2
    w1, w2 = q_dot # omega 1 and omega 2
    T1 = 0.5 * m1 * (l1 * w1)**2
    T2 = 0.5 * m2 * ((l1 * w1)**2 + (l2 * w2)**2 + 2 * l1 * l2 * w1 * w2 * torch.cos(t1 - t2))
    T = T1 + T2
    y1 = -l1 * torch.cos(t1)
    y2 = y1 - l2 * torch.cos(t2)
    V = m1 * g * y1 + m2 * g * y2
    
    return T - V

def equation_of_motion(lagrangian, state, t=None):
    n = state.shape[0]//2 # Get number of generalised coords
    state_tensor = torch.autograd.Variable(state, requires_grad=True)
    q, q_t = torch.split(state_tensor, 2)

    A = torch.inverse(hessian(lagrangian, state_tensor, create_graph=True))[n:, n:]
    B = jacobian(lagrangian, state_tensor, create_graph=True)[:n]
    C = hessian(lagrangian, state_tensor, create_graph=True)[n:, :n]
    
    q_tt = A @ (B - C @ q_t)
    
    res = torch.cat([q_t,torch.squeeze(q_tt)])

    return res

state = torch.tensor([3*np.pi/7, 3*np.pi/4, 0, 0])
q, q_t = torch.split(state, 2)
L_test = partial(lagrangian, m1=1., m2=1., l1=1., l2=1., g=9.8)(q, q_t)
equation_of_motion(L_test, state)
</code></pre>
<p>Thank you.</p>

ptrblck: <p>Could you check your imports and variable names, as based on the error message it seems you might have overridden the <code>hessian</code> method with a tensor using the same name.</p>
PyTorch and GLIBC compatibility error after upgrading JetPack to 4.5

janick187: <p>A few days ago we upgraded JetPack to <code>v4.5</code> on our Xavier Ubuntu 18.04 aarch64 machine. After installing PyTorch <code>v1.7.0</code> with torchvision <code>v8.1.0</code> I receive the following error when running <code>import torch</code> in Python <code>v3.6.9</code></p>
<pre><code class="lang-auto">ImportError: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /home/venv/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
</code></pre>
<p>Currently there is GLIBC <code>v2.27</code> installed.</p>
<p>Do you know how can I solve this problem? I read online that upgrading GLIBC is not my best option as this requires a lot of effort and some risk as may other aprty of my system depend on GLIBC <code>v2.27</code> .</p>

discort: <p>Just got the same issue on jetpack 4.4</p>
<p><a class="mention" href="/u/janick187">@janick187</a><br>
did you find a solution?</p>

janick187: <p>Hi mate,<br>
try to run Pytorch in a docker-container. This worked for me.</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://aws1.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="16" height="16">
      <a href="https://forums.developer.nvidia.com/t/pytorch-and-glibc-compatibility-error-after-upgrading-jetpack-to-4-5/176883/5" target="_blank" rel="noopener nofollow ugc" title="07:01AM - 11 May 2021">NVIDIA Developer Forums – 11 May 21</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:133/23;"><img src="https://aws1.discourse-cdn.com/nvidia/original/2X/8/8f17cc8f1a724d6ecea8a197a267ec8a05ef1490.png" class="thumbnail" width="133" height="23"></div>

<h3><a href="https://forums.developer.nvidia.com/t/pytorch-and-glibc-compatibility-error-after-upgrading-jetpack-to-4-5/176883/5" target="_blank" rel="noopener nofollow ugc">PyTorch and GLIBC compatibility error after upgrading JetPack to 4.5</a></h3>

<p>Hi @dusty_nv  Thanks a lot. With the docker container it is working like charm.  Just one additional question.  How can I save this docker image locally? Because additionally to pytorch I need to install some other dependencies and it would be messy...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
Question about CTC gradient

vadimkantorov: <p>In PyTorch CTC gradient takes log-probabilities as input (after log_softmax operation).</p>
<p>It seems that the CTC gradient formula in PyTorch (<a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/LossCTC.cpp#L302" rel="nofollow noopener">cpu version for simplicity</a>) refers to and seems to be implementing eq. 16 from the <a href="https://www.cs.toronto.edu/~graves/icml_2006.pdf" rel="nofollow noopener">original CTC paper</a>.</p>
<p>Eq. 16 seems to compute gradients wrt logits, not log_probs, and PyTorch must compute gradients wrt log_probs, so it’s strange if they are using the same equation.</p>
<p>This mismatch confuses me, I must be missing something obvious because gradcheck passes well, so the formula in PyTorch must be correct…</p>

tom: <p>At the risk of being the CTC person that I don’t want to be:</p>
<p>The key that the paper is not using log space, so we are not interested in the derivative by y, but by log y. With this, we can remind ourselves that is that log probs are logits (i.e. <code>x.log_softmax(dim=1) == x.log_softmax(dim=1).log_softmax(dim=1) </code>), so the derivative w.r.t. log y will be the one w.r.t. u.</p>
<p>The worst thing that happens is that you project the gradient onto the tangent space of the space (not a manifold, but close) of log probs twice (in CTC loss backward and then again in log_softmax backward). Is there a speedup to be had? In theory, yes, because the backward of the log_softmax you’ll be doing in PyTorch will compute the mean that we know is zero. But will we save from not doing it in the ctc backward? I haven’t thought about it much, but I expect not much, as we would need to specify ctc to take logits and take the logsoftmax in the ctc forward (but then we don’t need the backward).</p>
<p>Best regards</p>
<p>Thomas</p>

vadimkantorov: <p><a class="mention" href="/u/tom">@tom</a> Thank you very much! Now this complex situation is clearer!</p>

hzhwcmhf: <p>I also find the ctc backward is not consistent with the forward. It is kind of annoying that I find my network cannot converged, just because I do not use the log probability directly from the log_softmax function. I’d like to share a workaround if someone need the “true” gradient. <a href="https://github.com/pytorch/pytorch/issues/52241#issuecomment-841242760" class="inline-onebox" rel="noopener nofollow ugc">CTCLoss gradient is incorrect · Issue #52241 · pytorch/pytorch · GitHub</a></p>
Interpolate a tensor dimension from x to nx

wxystudio: <p>suppose a tensor shape [x, y, z …], I want to interpolate it to [nx, y, z …], every value in dim 0 repeat n times.<br>
For example, I have tensor[2, 3] =<br>
[[1, 2, 3],<br>
[4, 5, 6]]<br>
I want to interpolate it to tensor[6, 3], which is<br>
[[1, 2, 3],<br>
[1, 2, 3],<br>
[1, 2, 3],<br>
[4, 5, 6],<br>
[4, 5, 6],<br>
[4, 5, 6]]<br>
every value in dim 0 repeat 3 times, how to achieve it?</p>

pascal_notsawo: <p>Thanks to <a href="https://discuss.pytorch.org/t/repeat-examples-along-batch-dimension/36217/7">this post (Repeat examples along batch dimension)</a>, I used <a href="https://pytorch.org/docs/stable/generated/torch.repeat_interleave.html" rel="noopener nofollow ugc">torch.repeat_interleave</a> to do it.</p>
<pre><code class="lang-auto">import torch
torch.manual_seed(0)

shape = x, y = 2, 3
"""
or : 
  - shape = x, y, z = _, _, _, ...
  - shape = x, y, z, t, ... = _, _, _, _, ...
  ...
"""
n = 3

t = torch.rand(shape)
"""
tensor([[0.4963, 0.7682, 0.0885],
        [0.1320, 0.3074, 0.6341]])
"""

t_interpolate = torch.repeat_interleave(t, repeats=n, dim=0)
"""
tensor([[0.4963, 0.7682, 0.0885],
        [0.4963, 0.7682, 0.0885],
        [0.4963, 0.7682, 0.0885],
        [0.1320, 0.3074, 0.6341],
        [0.1320, 0.3074, 0.6341],
        [0.1320, 0.3074, 0.6341]])
"""

assert t_interpolate.size() == torch.Size([n * shape[0], *shape[1:]])
</code></pre>
View() and reshape()

hpf: <p>I learned something about the difference between <strong>view()</strong> and <strong>reshape()</strong>.</p>
<p>The <strong>view()</strong> does not change the original data stored. But <strong>reshape()</strong> may change the original data (when the original data is not continuous),<strong>reshape()</strong> may create a new memory space for the data</p>
<p>My doubt is whether the use of <strong>reshape()</strong> in RNN, CNN or other networks will affect the back propagation of errors, and affecting the final result？</p>

JuanFMontesinos: <p>It just make the computations less efficient.<br>
For example, RNN usually take temporal dimensions 1st because blocks are contiguous from left to right.<br>
If you slice through other dimensions, you are taking parts of different memory blocks.</p>

hpf: <p>thank you so much. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>
Transfer Learning - Image Deblurring: State of the Art

Ruffybeo: <p>Hi everyone!</p>
<p>I’m a newbie in Pytorch and came in contact with it through a university project. In this project, we received a dataset with microscopic images. Each observation/image in the dataset has two types: a blurry image (motion blur) and a sharp image. The goal of the project is to deblur the blurry images in the best way possible so that they are quite similar to their sharp counterpart.</p>
<p>I was able to set up a pipeline that includes data loading, data handling, preprocessing and am initial training and evaluation with a simple CNN for deblurring. Now, we got the task to use transfer learning to improve our results. Since then I"m really struggling with the project. I understand the basics of transfer learning (including how to use pre-trained models like VGG16, Resnet etc.). But when I dig deeper into the project, the following questions came to me:</p>
<ol>
<li>I suppose that I can’t just let a model such as VGG16 do all the deblurring because it is trained for classification tasks, right?</li>
<li>When I searched for papers of this topic, I found architectures like the ESRGAN(<a href="https://github.com/xinntao/ESRGAN" rel="noopener nofollow ugc">link</a>), Deblurganv2(<a href="https://github.com/VITA-Group/DeblurGANv2" rel="noopener nofollow ugc">link</a>) or SRCNN.  For each of them, I also found pre-trained models on Github. But except for the latter, I could not get them to work. I can load in the model but then I’m stuck in the part where I have to write a class that contains the architecture of the models. The codes in the repositories are so connected with other files in there that it is impossible to just write down the architecture (at least for me). Does anybody know a way to use these pre-trained models?</li>
<li>Are there any other important architectures that I’m missing? During my research, the most prominent architectures that I found were built on CNNs and GANs. But I’m not sure if I maybe miss some state of the art architecture for motion deblurring?</li>
</ol>
<p>I know that is a lot to ask and read but I would be grateful for any input :).</p>

pchandrasekaran: <p>Are you looking to complete the project <strong>ONLY</strong> with pytorch?</p>
<p>If yes, I’m sorry, I can’t help. Same problem as you, can’t figure how to get the pretrained models to work.</p>
<p>If no, then you can use the OpenVINO model for deblurring. If you decide to go with it, know that is a bit finnicky to setup, so let me know if you need any help.</p>
How to get input data from a dataloader?

nikin: <p>Hi guys,<br>
I am relatively new to neural networks and for a project I am currently working on I am given a train_dataloader and I need to get the input data. Can somebody please help me on this?<br>
I know that you can get the labels with train_dataloader.dataset.targets but how do I get the features?</p>
<p>Thank you so much!</p>

deepfailure: <p>To train what is usually done is to iterate over the dataloader such as:</p>
<pre><code class="lang-auto">for i, (x, y) in enumerate(train_loader)
</code></pre>
<p>Where i is the iteration, x are the features and y is the target.</p>

nikin: <p>Thanks, but I need the whole input data because I need to do some preprocessing and I would prefer not to get it with a for loop for efficiency reasons</p>
Is balancing mini-batches over all the epoch required?

twister9458: <p>Hello all;</p>
<p>I’ve an unbalanced dataset for a 3 classes output.<br>
I managed to have well stratified train and test sets.</p>
<p>My question please: Using WeightedRandomSampler gives me bad results rather than without using it. (74% vs 91% accuracies)</p>
<p>Is there any logical explanation for this ?</p>
<p>Last question: assuming all my dataset was well balanced, I’m I required to have balanced mini-batches during the epoch ? Can I have for example first mini-batches with only 2 classes, and the last ones with the last class ? Is this okay for the training ? note that this question isn’t pertaining to my use.</p>
<p>Thank you very much,<br>
Habib</p>

ptrblck: <aside class="quote no-group" data-username="twister9458" data-post="1" data-topic="121164">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/db5fbb/48.png" class="avatar"> twister9458:</div>
<blockquote>
<p>My question please: Using WeightedRandomSampler gives me bad results rather than without using it. (74% vs 91% accuracies)</p>
<p>Is there any logical explanation for this ?</p>
</blockquote>
</aside>
<p>You might be running into the <a href="https://en.wikipedia.org/wiki/Accuracy_paradox">Accuracy Paradox</a>, which explains that the accuracy might be a misleading metric for imbalanced classification use cases.<br>
E.g. in case the majority class occurs in 91% of all cases, your model might only predict this single class to achieve the accuracy of 91%. While the model would be not very useful, the accuracy “looks good”.<br>
Balancing the dataset might thus reduce the accuracy, but increase other metrics such as the F1 score.</p>

twister9458: <p>Makes sense.</p>
<p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for strongly supporting this forum.</p>
Resnet18_test and training loss vibrate greatly

celine_celine: <p>Hello, everyone!<br>
I am training a 2-class resnet18 network using images(about 4000 pieces) from cocodataset.<br>
I find all test and training loss vibrate greatly.<br>
It looks like something wrong in training parameters.<br>
I have changed batch size 64 and 128, also pretrained True and false.<br>
But nothing changed.<br>
Please help me.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fd9275a937d07ed2e73f87e6840339c543e9e13.png" data-download-href="https://discuss.pytorch.org/uploads/default/7fd9275a937d07ed2e73f87e6840339c543e9e13" title="4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fd9275a937d07ed2e73f87e6840339c543e9e13_2_690x311.png" alt="4" data-base62-sha1="ieZT5eDsXZQgmv7CvYUtv7aIWeD" width="690" height="311" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fd9275a937d07ed2e73f87e6840339c543e9e13_2_690x311.png, https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fd9275a937d07ed2e73f87e6840339c543e9e13.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fd9275a937d07ed2e73f87e6840339c543e9e13.png 2x" data-dominant-color="FBF7F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">4</span><span class="informations">1024×462 68.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Dropconnect result warning in rnn

kongmuqiuren: <p>I am trying to use drop connect in rnn, but just to receive a warning:</p>
<pre><code class="lang-auto">‘UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greately increasing memory usage. To compact weights again call flatten_parameters()’
</code></pre>
<p>here is the weight dropout code:</p>
<pre><code class="lang-auto">def _weight_drop(module, weights, dropout):
    """
    Thanks to pytorchnlp, here is the LICENSE:https://hub.fastgit.org/PetrochukM/PyTorch-NLP/blob/master/LICENSE
    """

    for name_w in weights:
        w = getattr(module, name_w)
        del module._parameters[name_w]
        module.register_parameter(name_w + '_raw', nn.Parameter(w))

    original_module_forward = module.forward

    def forward(*args, **kwargs):
        for name_w in weights:
            raw_w = getattr(module, name_w + '_raw')
            w = F.dropout(raw_w, p=dropout, training=module.training)
            setattr(module, name_w, w)

        return original_module_forward(*args, **kwargs)

    setattr(module, 'forward', forward)
</code></pre>
<p>And in rnn module, I write something like this:</p>
<pre><code class="lang-auto">        if wdrop:
            weights = ["weight_hh_l" + str(i) for i in range(self.n_layers)]
            if bidirectional:
                weights_reverse = ["weight_hh_l" +
                                   str(i) + "_reverse" for i in range(self.n_layers)]
                weights = weights+weights_reverse
                _weight_drop(self.rnn, weights, wdrop)
</code></pre>
<p>Notice: It’s ok to run my model(verified), but just annoyed for me to see the warning! I have googled this problem and find <a href="https://stackoverflow.com/questions/53231571/what-does-flatten-parameters-do" rel="noopener nofollow ugc">flatten parameters</a>, but after add "rnn.flatten_parameters() " still can not avoid this warning.</p>
<p>So, my question is why it happens, and how to eliminate it?</p>
VOCDetection - error, NO route to host

111490: <p>I used torch.utils.datasets.VOCDetection, but there was error message<br>
how can i address this error?</p>
<p>There was no error when I used it a month ago.</p>
<p>train_ds = VOCDetection(path2data, year=‘2007’, image_set=‘train’, download=True</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/526c4174e9dd2be3abf5d4b010413405f9b447a6.png" data-download-href="https://discuss.pytorch.org/uploads/default/526c4174e9dd2be3abf5d4b010413405f9b447a6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/2/526c4174e9dd2be3abf5d4b010413405f9b447a6.png" alt="image" data-base62-sha1="bL95xmgMlxvED4aT3ZezUSBTNiK" width="690" height="377" data-dominant-color="F5F5F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">768×420 8.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>It seems that the download fails. It seems that <a href="http://host.robots.ox.ac.uk/">http://host.robots.ox.ac.uk/</a> is down, at least I also cannot access it.</p>
Setup the model for gradient calculation

sebko_iic: <p>Is it possible to setup a network consisting of only standard layers (conv2d, batchnorm2d, relu, avgpoling, fc) for the backward pass.<br>
Assuming I already done a forward pass but I did it with torch.no_grad() but I have saved the output of each layer. Is it possible to setup each layers with the previous output so that I can just call the backward function from PyTorch?</p>
How can I do batch-wise a matrix vector row-wise multiplication

ECL: <p>If I have a matrix <code>M</code> of size <code>(d1, d2)</code> and a vector <code>V</code> of size <code>d2</code>, doing <code>M*V</code> gives me an output <code>OUT</code> of size <code>(d1, d2)</code>, where each row of <code>M</code> has been multiplied by <code>V</code>.</p>
<p>I need to do the same thing batch-wise, where the matrix <code>M</code> is fixed and I have a batch of <code>dB</code> vectors.</p>
<p>In practice, given a tensor <code>M</code> of size <code>(d1, d2)</code> and a tensor <code>V</code> of size <code>(dB, d2)</code>, I need to get as output <code>OUT</code> a tensor of size <code>(dB, d1, d2)</code>, so that <code>OUT[i] = M*V[i]</code>.</p>
<p>How can I efficiently get it?</p>

eqy: <p>I think you can get the desired behavior by explicitly inserting the dimension that was implicitly inserted for the single vector case via <code>unqueeze(1)</code> on your “batched” vector.<br>
Example:</p>
<pre><code class="lang-auto">import torch

d1 = 16
d2 = 32
dB = 64

m = torch.randn(d1, d2)
v = torch.randn(dB, d2)
out = torch.empty(dB, d1, d2)
for i in range(dB):
    out[i, :, :] = m*v[i]
print(out.shape)
v2 = v.unsqueeze(1)
out2 = m*v2
print(out2.shape)
print(torch.allclose(out, out2))
</code></pre>

huahuanZ: <p>I think <code>torch.einsum</code> can simply do that. But I’m not sure whether it’s efficient or not.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; A = torch.randn((3,4))
&gt;&gt;&gt; B = torch.randn((16,4))
&gt;&gt;&gt; C = torch.einsum("ij,bj-&gt;bij", (A, B))
&gt;&gt;&gt; C.size()
torch.Size([16, 3, 4])
&gt;&gt;&gt; C[0]-A*B[0]  # for testing the correctness
tensor([[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]])
&gt;&gt;&gt; C[5]-A*B[5]
tensor([[0., 0., 0., 0.],
        [0., 0., 0., 0.],
        [0., 0., 0., 0.]])
</code></pre>
Custom 3d convolutions with 2d "internal convolutions"

uh123: <p>Usually in a convolution we just multiply the kernel with the input and return the scalar, and then move the kernel and repeat the process. The problem I have is that instead of returning a scalar I have an “internal” 2d plane to convolve over, at each 3d point. So the 3d convolution has to return, instead of a scalar, a 2d convolution result.</p>
<p>I am not sure how to implement something like this. A good starting step for me would be if I could efficiently extract 3x3x3 chunks of the input in the same way that pytorch does for regular 3d convolutions. I have tried looking at conv3d in torch/nn/quantized/functional.py but cant seem to find the part the actual convolutions are being taken.</p>

ptrblck: <p>To create these “patches” you could use <a href="https://pytorch.org/docs/stable/generated/torch.nn.Unfold.html#torch.nn.Unfold"><code>nn.Unfold</code></a>, which also shows the manual conv operation in its example.</p>
Error after adding layer to downloaded model

nikoskot: <p>Hi,<br>
I am using the DeepLabV3_ResNet50 model for a project and i want to build a classifier with this network. So I tried to add a Flatten layer at the end of the network using this command <code>model2.classifier.add_module('fl', torch.nn.Flatten())</code>, in order to add later some fully connected layers. But after adding this layer, when I try to pass a dummy input <code>input = torch.ones(1, 3, 112, 112)</code>, I get the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/usr/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "&lt;input&gt;", line 1, in &lt;module&gt;
  File "/home/spbtu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/spbtu/.local/lib/python3.8/site-packages/torchvision/models/segmentation/_utils.py", line 25, in forward
    x = F.interpolate(x, size=input_shape, mode='bilinear', align_corners=False)
  File "/home/spbtu/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 3079, in interpolate
    raise ValueError('size shape must match input shape. '
ValueError: size shape must match input shape. Input is 0D, size is 2
</code></pre>
<p>Is something wrong with the way I add the flattening layer?</p>

ptrblck: <p><code>nn.Flatten</code> would use the default arguments:</p>
<pre><code class="lang-python">start_dim=1, end_dim=-1
</code></pre>
<p>so the batch dimension (dim0) should stay the same while all additional dimensions would be flattened.</p>
<p>Based on the error message it seems you are trying to apply a 2D interpolation on this flattened tensor, which won’t work:</p>
<pre><code class="lang-python">x = torch.randn(1, 3, 4, 4)
out = F.interpolate(x, size=(10, 10)) # works

y = x.view(x.size(0), -1)
out = F.interpolate(y, size=(10, 10)) # error
&gt; ValueError: size shape must match input shape. Input is 0D, size is 2

out = F.interpolate(torch.nn.Flatten()(x), size=(10, 10)) # error
&gt; ValueError: size shape must match input shape. Input is 0D, size is 2
</code></pre>
<p>since you would need to use a 3-, 4, or 5-dimensional tensor as given in the code snippet (<code>[batch_size, channels, seq_len]</code>, <code>[batch_size, channels, height, width]</code>, or <code>[batch_size, channels, depth, height, width]</code>).</p>
How to dealt with N=0 when using `smooth_L1 Loss`?

Igo312: <p>Cause I use a mask to filter the input and target. Sometimes the mask will be empty so that input and target will be empty too.</p>
<p>When it happen, if using <code>nn.functional.smooth_l1_loss</code> it will retuan a <strong>nan</strong>. What I fix it is return a zero. Any better way to fix it?</p>
<p>here’s my regression loss</p>
<pre><code class="lang-auto">    def regr_loss(self, regr, gt_regr, mask):
        mask = mask.gt(0)
        num = mask.float().sum()
        if num == 0:
            return Variable(torch.tensor(0).cuda())
        mask = mask.unsqueeze(1).expand_as(gt_regr)
        regr = regr[mask]
        gt_regr = gt_regr[mask]

        regr_loss = nn.functional.smooth_l1_loss(regr, gt_regr)
        regr_loss = regr_loss / (num + 1e-4)
        return regr_loss
</code></pre>
Trying to implement a simple federated agregation function is not working

luis_alfonso_Palomin: <p>I’m trying to simulate federated learning with a simple approach: I have a “local model” and some “workers” that have their own models (at the beginning just a copy from local model) and data. The data should be private so only the models can have access through ‘worker.model’</p>
<p>Every epoch after all workers are done with the training on own data a federated_aggregation function is called which calculates the average of all worker model parameters and save it to the specific local model parameter:</p>
<pre><code class="lang-auto">def federated_aggregation(local_model, workers):

    with torch.no_grad():

        for name, local_parameter in local_model.named_parameters():

            parameter_stack = torch.zeros_like(local_parameter)

            for worker in workers:
                
                parameter_stack += worker.model.state_dict()[name].clone().detach()

            parameter_stack /= len(workers)
            local_parameter.set_(parameter_stack)
</code></pre>
<p>The thing is, when evaluating the local model, this doesn’t reflect the performance it should get after the training of the workers. A simple test to visualize this was to run a test with just one worker:</p>
<p>Performance of just one worker (training recall, validation recall) left, right the validation recall of local model:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/7/8706423fd9f4e3773fddb0e730fbfeac8e1a58b2.png" data-download-href="https://discuss.pytorch.org/uploads/default/8706423fd9f4e3773fddb0e730fbfeac8e1a58b2" title="summary"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/8706423fd9f4e3773fddb0e730fbfeac8e1a58b2_2_690x190.png" alt="summary" data-base62-sha1="jgtROcKqMi1D0yKR1y5nxjNYiEG" width="690" height="190" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/8706423fd9f4e3773fddb0e730fbfeac8e1a58b2_2_690x190.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/7/8706423fd9f4e3773fddb0e730fbfeac8e1a58b2_2_1035x285.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/7/8706423fd9f4e3773fddb0e730fbfeac8e1a58b2.png 2x" data-dominant-color="B8BABB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">summary</span><span class="informations">1180×326 40.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any ideas what is the cause for this mismatch, I’ve tried different ways to update de local model parameter but I’m stuck in this behavior</p>
Tensor Manipulation for Padding

VIGNESHinZONE: <p>I’m working with graph neural networks and it’s important for my work to perform some padding. Here is the problem -</p>
<p>I have 3 graphs [g1, g2, g3] and each with 1,2,3 nodes respectively. I want to aggregate these nodes such that they are padded with a fixed size. Here is the example</p>
<pre><code class="lang-auto">all_nodes_feature = torch.tensor[ 
[1, 1],
[2, 2],
[2, 2],
[3, 3],
[3, 3],
[3, 3] ]

number_of_nodes_in_each_graph = torch.tensor([1, 2, 3])

fixed_length = 3

resulting_output = [ 
[ [1,1], [0, 0], [0,0] ],
[ [2,2], [2, 2], [0,0] ],
[ [3,3], [3, 3], [3,3] ]
]
</code></pre>

Unity05: <p>Hi,</p>
<p>you could firstly split the <code>all_nodes_feature</code> tensor to then pad them as sequences. Be aware however that in this case <code>number_of_nodes_in_each_graph</code> should be a normal list of integers.<br>
So you would have:</p>
<pre><code class="lang-auto">all_nodes_feature = torch.tensor([[1, 1],
                                 [2, 2],
                                 [2, 2],
                                 [3, 3],
                                 [3, 3],
                                 [3, 3]])

number_of_nodes_in_each_graph = [1, 2, 3]

all_nodes_feature_tensor_list = torch.split(
    tensor=all_nodes_feature, 
    split_size_or_sections=number_of_nodes_in_each_graph
)

all_nodes_feature_padded = torch.nn.utils.rnn.pad_sequence(
    sequences=all_nodes_feature_tensor_list, 
    batch_first=True, 
    padding_value=0
)
</code></pre>
<p>I hope I could help you with that.<br>
Regards,<br>
Unity05</p>

VIGNESHinZONE: <p>Yup it worked, thanks a lot <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Restore the original image from ViT embedding

xincz: <p>Let’s say I have an PIL Image of shape (224, 224, 3). I want to use Vision Transformer so that I can get the sequential embeddings of the image, which is of shape (1, 197, 768). Then, I’d like to restore the original image dimensions from the Tensor of shape (1, 197, 768), namely (224, 224, 3). I want every pixel to be at exactly the same position as they did in the original image. How do I achieve this? This is something I have tried:</p>
<pre data-code-wrap="python"><code class="lang-python">em = res.squeeze()  # (1, 197, 768) -&gt; (197, 768)
cls_token, emb = em[0], em[1:]  # exclude the class token
emb = emb.detach().numpy()  # convert torch.Tensor to numpy.ndarray
einops.rearrange(emb, '(l1 l2) (c h w) -&gt; (h l1) (w l2) c', l2=14, h=16, c=3)  # use einops' operation
</code></pre>
<p>The result:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f106336191bcfd1cf62f5fa841e9b6a17c80449b.png" alt="DeepinScreenshot_select-area_20210514110833" data-base62-sha1="yocm1TLEI7CE9JvEZv1daWL1XRN" width="288" height="286"></p>
Construct graph from Neural network

szhang1103: <p>I  am wondering How we could construct a graph from neural network where layer are nodes.</p>
<p>I have some vague idea of how to do this. For example, we can use hooks to record the layer and their connection, but this method seems to only work for sequential model, when we have skip connections, this does not work well.</p>
<p>I have also seen people using onnx and traces to construct a trace graph, but in that case, the node are onnx  operation instead of pytorch module.</p>
Subsampling rows speedups? CPU vs GPU

GeoffNN: <p>I’m working on a stochastic method acting row wise on a tensor. I’ve realized some strange behavior on CPU vs GPU.</p>
<p>Suppose I use an MSE loss:</p>
<pre><code class="lang-auto">sqloss = torch.nn.MSELoss()
</code></pre>
<p>Then, I get the following time results. <code>Z</code> requires a gradient, <code>M</code> does not. For now, I’m sampling <code>idx</code> at each iteration. I’ve also tried to put <code>Z</code> and <code>M</code> into a <code>torch.utils.data.TensorDataset(Z, M)</code>, to no avail.</p>
<pre><code class="lang-auto">Z = torch.rand(1000,1000)
Z.requires_grad_(True)
M = torch.rand(1000, 1000)
idx = torch.randint(Z.size(0), size=(10,))
</code></pre>
<p>Replacing <code>sqloss</code> from above by the MSELoss:<br>
<code>sqloss = MSEloss()</code></p>
<ul>
<li>GPU</li>
</ul>
<pre><code class="lang-auto">In [6]: %timeit sqloss(Z, M)
49 µs ± 7.4 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [7]: %timeit sqloss(Z[idx], M[idx])
222 µs ± 32.3 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
</code></pre>
<ul>
<li>CPU:</li>
</ul>
<pre><code class="lang-auto">In [10]: %timeit sqloss(Z, M)
612 µs ± 167 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [11]: %timeit sqloss(Z[idx], M[idx])
52.9 µs ± 2.25 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
</code></pre>
<p>This is due to the indexing:</p>
<ul>
<li>GPU:</li>
</ul>
<pre><code class="lang-auto">In [21]: %timeit Z[idx]
96.3 µs ± 15.7 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
</code></pre>
<ul>
<li>CPU:</li>
</ul>
<pre><code class="lang-auto">In [25]: %timeit Z[idx]
13 µs ± 2.63 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)
</code></pre>
<p>Is this behavior expected? Why is the subsampled loss much faster than the full loss on CPU but not on GPU ? Is there anyway to speed this up?</p>
About saving image methods difference

Florentino: <p>Hi, I discover that when I use plt.savefig (cmap=“gray”)and torchvision.save_image to save a identical element but different dtype (ndarray and torch.Tensor) in gray scale, but the result in brightness is very different, how can I tune or is there any different algorithms behind two functions? Thanks<br>
Below is my output feature map(above right one) and below( I know it looks too dark, but it indeed has the pattern when you look zoom in)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/2/220192d4676f4834481887809f61f452dbaea237.png" alt="image" data-base62-sha1="4QPBwQgqQ49D1MYzh7phiqWiFEj" width="640" height="480"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fc022447f95b1bfd9447bb04ffc61b654f38d56.png" alt="image" data-base62-sha1="2fkO1UolZBQyu71YfQfluU7H54O" width="52" height="52"></p>
Create a custom dataset using a sliding window from multiple sets of time series data

semaj: <p>Hello, long time listener first time caller!</p>
<p>I’m trying to create custom a dataset from a set of time series .csv’s with accompanying <em>ground truth</em> .csv’s. Let’s say my folder structure looks like:</p>
<pre><code class="lang-auto">root/
    - action1/
        -sync/
            -data1.csv
            -data2.csv
            -truth1.csv
            -truth2.csv
            - ...
    - action2/
        -sync/
            -data1.csv
            -data2.csv
            -truth1.csv
            -truth2.csv
            - ...

</code></pre>
<p>I would like to take windows from each .csv (e.g. data1 and truth1) as my x and y to feed into my NN. I can write a function that stores the path to each .csv and stores the accompanying length. My issue is how to write the custom dataset class - if it is possible?</p>
<p>All the .csv’s are of unique length so I would like to interate over each .csv at a time and take len(data[i]) - windowlength sequences from each.</p>
<p>When defining the <strong>len</strong>, is this the TOTAL number of windows from all the data?</p>
<p>Any help would be greatly GREATLY appreciated. Thankyou:)</p>
Cosine Similarity with two tensors of different dimensions

Yunchao_Liu: <p>Hello, I have two tensors. Tensor A is of size (n,d), and tensor B is of size (n, k, d). I want to calculate the cosine similarity for the first row of tensor A to each row of the first layer in Tensor B. Then to calculate the cosine similarity for the second row of tensor A to each row of the second layer in Tensor B. So on so forth as illustrated below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90097bd03706a325c94c697a034d0fddf5583c44.png" data-download-href="https://discuss.pytorch.org/uploads/default/90097bd03706a325c94c697a034d0fddf5583c44" title="cos_simi"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/0/90097bd03706a325c94c697a034d0fddf5583c44.png" alt="cos_simi" data-base62-sha1="kyd4BQAW9Gt53mLbWJWTcB0H2vO" width="623" height="500" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cos_simi</span><span class="informations">921×739 11 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I cannot think of a good way of doing it rather than using a for-loop. Can someone help me?</p>

Yunchao_Liu: <p>never mind. I solved this by expanding tensor A to be of the same size of tensor B.</p>
How to write tree-lstm in pytorch?

Demi_Guo: <p>Hi all,</p>
<p>I’m wondering how should I write a tree-lstm in pytorch? Is there any particular functionalities I should look more into? Is there any similar examples I should look at? What is the roughly outline of the implementation?</p>
<p>Thanks!</p>
<p>Best,<br>
Demi</p>

DiffEverything: <p>You can have a look at this implementation - <a href="https://gist.github.com/wolet/1b49c03968b2c83897a4a15c78980b18" rel="nofollow noopener">https://gist.github.com/wolet/1b49c03968b2c83897a4a15c78980b18</a></p>

jekbradbury: <p>Or this one, if you want a partially-batched approach like the SPINN paper: <a href="https://github.com/jekbradbury/examples/blob/spinn/snli/spinn.py" rel="nofollow noopener">https://github.com/jekbradbury/examples/blob/spinn/snli/spinn.py</a></p>

ThaiThien: <p>I already implement the pytorch version here <a href="https://github.com/ttpro1995/TreeLSTMSentiment" rel="nofollow noopener">https://github.com/ttpro1995/TreeLSTMSentiment</a><br>
Which I convert from torch code <a href="https://github.com/stanfordnlp/treelstm" rel="nofollow noopener">https://github.com/stanfordnlp/treelstm</a></p>

ksolaiman: <p>In your implementation of the BinaryTreeLSTM,</p>
<ol>
<li>
<p>Can you explain the leaf / base condition - self.ox just passes through a linear layer, that is understandable since there is no hidden state for leafs, but why is there no weight params for input, update or forget gating, and why is cell state just passed through a linear layer?</p>
</li>
<li>
<p>And also for non-leaf nodes, you are completely ignoring passing the input through a linear layer, for all the gating units. Is there an explanation for that? In ChildSum, you have weight parameters for x_j, why not in n-ary lstm ?</p>
</li>
</ol>
<pre><code class="lang-auto">self.ix = nn.Linear(self.in_dim,self.mem_dim)
</code></pre>

Suzyahyah: <p>A <a href="https://suzyahyah.github.io/pytorch/2019/08/11/TreeLSTMS.html" rel="noopener nofollow ugc">implementation with easy-first parsing, with explanations</a></p>
<p>Tree LSTMs are conceptually straightforward extension of RNN-LSTMs. Here’s what we need:</p>
<ol>
<li>A parser (takes in a sentence and outputs the parse tree structure).</li>
</ol>
<ul>
<li>Stanford Core NLP/your own parser</li>
</ul>
<ol start="2">
<li>
<p>Given the parse tree structure which implicitly contains how the word units should be progressively combined, convert this into a <em>series of instructions</em> which explicitly describes how the nodes should be combined.</p>
</li>
<li>
<p>Write a RNN that takes in a series of instructions on how to combine a list of inputs.</p>
</li>
</ol>

Brando_Miranda: <p>perhaps using a well tested library for graphs is the way to go: <a href="https://docs.dgl.ai/en/0.4.x/tutorials/models/2_small_graph/3_tree-lstm.html" class="inline-onebox" rel="noopener nofollow ugc">Tutorial: Tree-LSTM in DGL — DGL 0.4.3post2 documentation</a></p>
CosineSimilarity > 1

Yunchao_Liu: <p>Hello, I’m trying to understand the CosineSimilarity for tensors with different dimensions. I don’t quite understand the reason why the output is &gt;1. Shouldn’t cosine similarity always be between -1 to 1?</p>
<p>from torch.nn import CosineSimilarity as cs</p>
<p>f = torch.tensor([[3,2]], dtype = torch.float)<br>
g = torch.tensor([[3,2],[1,2]], dtype = torch.float)</p>
<p>cos = cs(dim =0,eps = 0.3)<br>
z = cos(d,e)<br>
print(z)</p>
<p>output: tensor([1.2649, 1.4142])</p>

ariG23498: <p>Hey <a class="mention" href="/u/yunchao_liu">@Yunchao_Liu</a><br>
I think here you would need to keep the dims of both the matrices the same. With reference to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.CosineSimilarity.html" rel="noopener nofollow ugc">docs</a> we need to have same shape matrices and do the cosine similarity on a said <code>dim</code>.</p>
<p>With that in mind, we will have a value from the range of <code>-1</code> to <code>1</code>.</p>

Yunchao_Liu: <p>Hi <a class="mention" href="/u/arig23498">@ariG23498</a>  Thanks!</p>
<p>Now I have another question since I need to implement calculating cosine similarity between tensors of different dimensions. This is the <a href="https://discuss.pytorch.org/t/cosine-similarity-with-two-tensors-of-different-dimensions/121087">problem description</a></p>
How to use gradient checkpointing on packed sequence RNN

kenenbek: <p>I have a batch of sequences that have a variable length. To save computation I used <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.rnn.PackedSequence.html#packedsequence" rel="noopener nofollow ugc">pack_padded_sequence</a> as following:</p>
<pre><code class="lang-auto">input = torch.nn.utils.rnn.pad_sequence(input, batch_first=True)
input = torch.nn.utils.rnn.pack_padded_sequence(input,
                                                batch_first=True,
                                                lengths=lengths)
</code></pre>
<p>Because sequences are long, I use gradient <a href="https://pytorch.org/docs/stable/checkpoint.html" rel="noopener nofollow ugc">checkpointing</a> to save memory</p>
<p><code>output, hiddens = cp.checkpoint(self.gru, *(input, hiddens, self.dummy_tensor))</code></p>
<p>As a result I have such error:</p>
<pre><code class="lang-auto">File ".../src/sequence_models/gru.py", line 86, in forward
    output, hiddens = cp.checkpoint(self.gru, *(input, hiddens, self.dummy_tensor))
  File ".../torch/utils/checkpoint.py", line 177, in checkpoint
    return CheckpointFunction.apply(function, preserve, *args)
TypeError: CheckpointFunctionBackward.forward: expected Tensor or tuple of Tensor (got PackedSequence) for return value 0
</code></pre>
<p>How can I handle it?</p>
Imagewoof: correct use of `transforms.Compose()` for testing?

Samue1: <p>I want to train a network on the <a href="https://github.com/fastai/imagenette" rel="noopener nofollow ugc">Imagewoof</a> dataset. I would like to know if I correctly use <code>transforms.Compose()</code> for testing my network.</p>
<p>Currently I use the following approach:</p>
<pre><code class="lang-auto">avg = (0.4914, 0.4822, 0.4465)
std = (0.2023, 0.1994, 0.2010)
stats = (avg, std)

train_transform = transforms.Compose([
    transforms.RandomResizedCrop(size=(128, 128)),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(*stats, inplace=True)])

test_transform = transforms.Compose([
    transforms.RandomResizedCrop(size=(128, 128)),
    transforms.ToTensor(),
    transforms.Normalize(*stats)])
</code></pre>
<p>However, I think it is not right to use <code>transforms.RandomResizedCrop(size=(128, 128))</code> for testing my network, because it randomly crops the image. If I want to compare my results to those from the official leaderboard, how do I have to configure <code>test_transform</code>?</p>

eqy: <p>I believe the standard approach is to use a <a href="https://pytorch.org/vision/stable/transforms.html#torchvision.transforms.CenterCrop" rel="noopener nofollow ugc">Center Crop</a> although different datasets may use different “rules” sometimes. Note that you may want to resize the images first (e.g., <a href="https://github.com/pytorch/examples/blob/01539f9eada34aef67ae7b3d674f30634c35f468/imagenet/main.py#L227" class="inline-onebox" rel="noopener nofollow ugc">examples/main.py at 01539f9eada34aef67ae7b3d674f30634c35f468 · pytorch/examples · GitHub</a>) before taking a center crop due to the variation in resolution of images.</p>

Samue1: <p>Using Center Crop seems like the most logical approach, thanks!</p>
<p>Do I need to resize the images, if I know, that their height and width is larger than 128 pixels?</p>

eqy: <p>For example, if a test image is 10000x10000 pixels, taking the 128x128 center crop before resizing to say (160x160 or 192x192) might result in a huge mismatch between the size of objects your model is used to seeing at training time and what it is now being tested on. Resizing to a known size before taking the center crop is a way to help alleviate this problem.</p>
<p>Overall, this issue is somewhat tricky to deal with even with resizing (see <a href="https://arxiv.org/abs/1906.06423" rel="noopener nofollow ugc">Touvron et. al</a> for more details).</p>
Ignore some op in torch.autograd.profiler/torch.profiler

111530: <p>Background: I am profiling my model to see the bottleneck. I am not interested in the dataloader, so ignoring its performance is ok for me.</p>
<p>I use the profiler as shown below:</p>
<pre><code class="lang-python">    with torch.autograd.profiler.profile(
        enabled=True,
        use_cuda=False,
        record_shapes=True,
        with_flops=True,
        profile_memory=True,
        use_cpu=True
    ) as prof:
        with torch.no_grad():
            for _ in range(args.nepochs):
                for i, test_batch in enumerate(test_ld):
                    X_test, y_test, z_test, T_test = test_batch
                    R_test = model_wrap(
                        model,
                        X_test,
                        y_test,
                        z_test,
                    )
</code></pre>
<p>And later I found the <code>enumerate(DataLoader)#_SingleProcessDataLoaderIter._... </code> occupies more than 90% in the breakdown, though I don’t care about it. It makes the log unreadable. How can I remove its information from the profiler?</p>