akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Output regression standardisation

frc: <p>Hi! I’ve just started coding with PyTorch and I’ve built a regression model of a ResNet that, given an image 1x128x128, outputs a tensor made by 3 values (tensor([1,1,3]). The network should therefore predict the 3 values of the tensor. Being two of them always way larger than the other (e.g. [512, 520, 100]) I was thinking to apply a standardisation to the targets. How can this be done in PyTorch?<br>
Hoping it’s clear what I mean. Thank you so much!</p>
Torch jit load model error,help!

c9412600: <p>we use pytorch version of 1.7.0+cu101 to turn torchscript，and I successfully loaded the torchscript model in torch version of 1.7.0+cu101 to inference, but when I use pytorch version of 1.8.1+cu101 to inference, there are some error<br>
<code>python3 torch_test_jit_1109.py  Traceback (most recent call last):   File "torch_test_jit_1109.py", line 35, in &lt;module&gt;     main(sys.argv[1], sys.argv[2])   File "torch_test_jit_1109.py", line 17, in main     e2e_model = torch.jit.load(script_model)   File "/data3/users/suziyi/Miniconda3/lib/python3.8/site-packages/torch/jit/_serialization.py", line 161, in load     cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files) RuntimeError: Class Namespace cannot be used as a value: Serialized   File "code/__torch__/torchaudio/sox_effects/sox_effects.py", line 5     effects: List[List[str]],     channels_first: bool=True) -&gt; Tuple[Tensor, int]:   in_signal = __torch__.torch.classes.torchaudio.TensorSignal.__new__(__torch__.torch.classes.torchaudio.TensorSignal)                                                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE   _0 = (in_signal).__init__(tensor, sample_rate, channels_first, )   out_signal = ops.torchaudio.sox_effects_apply_effects_tensor(in_signal, effects) 'apply_effects_tensor' is being compiled since it was called from 'SynthesizerTrn.forward' Serialized   File "code/__torch__/models_torchaudio_torchscript.py", line 41     sr: Tensor,     vol: Tensor) -&gt; Tensor:     _0 = __torch__.torchaudio.sox_effects.sox_effects.apply_effects_tensor     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE     _1 = torch.tensor([(torch.size(x))[-1]], dtype=None, device=None, requires_grad=False)     x_lengths = torch.to(_1, 4, False, False, None) </code><br>
we use torchaudio to change audio speed,volume and so on, please help me<br>
when we create torchscript the pytorch version is 1.7.0+cu101 , the torchaudio version is 0.7.0</p>
Mid_channels in U2NET

Tornike: <p>Hi, Doing Depth Estimation Task with U2NET.</p>
<p>I read the Paper but still don’t get it.</p>
<p>What are the mid_channels? why they use it?<br>
and why they use that in every conv Layer after first Conv.</p>
<pre><code class="lang-auto">class RSU7(nn.Module):#UNet07DRES(nn.Module):

    def __init__(self, in_ch=3, mid_ch=12, out_ch=3):
        super(RSU7,self).__init__()

        self.rebnconvin = REBNCONV(in_ch,out_ch,dirate=1)

        self.rebnconv1 = REBNCONV(out_ch,mid_ch,dirate=1)
        self.pool1 = nn.MaxPool2d(2,stride=2,ceil_mode=True)

        self.rebnconv2 = REBNCONV(mid_ch,mid_ch,dirate=1)
        self.pool2 = nn.MaxPool2d(2,stride=2,ceil_mode=True)

</code></pre>

ptrblck: <p>I don’t know what is mentioned in the paper, but based on the posted code snippet it seems to be an architecture choice which would simplify the model creation by defining the input, mid, and output channels only.</p>

Tornike: <p>Yes, makes sense.</p>
<p>Thanks for your reply.</p>
How to train pretrained Faster RCNN with oriented bounding boxes

Arda_Tumay: <p>I am trying to train pretrained faster rcnn model with oriented bounding boxes. Dataset I use is DOTA. All instances are annotated by oriented bounding boxes. Is there any way in pytorch to train faster rcnn with oriented bboxes? Or can anyone suggest me a way to do so?</p>
<p>Thanks for any help.</p>

Arda_Tumay: <p>Edit: As long as I understand, there are two different approaches to detect rotated objects in images. The first one is to use rotational rpn (RRPN) which is a modified RPN network that produces inclined proposals and anchors.  The second one is to use R2CNN which is a faster rcnn with an extra regression branch for the calculation of angle.</p>
<p>As I searched, pytorch does not have any built-in implementation for these networks. Moreover, I also couldn’t find proper and basic implementation details on the internet. So, is it easy to implement an extra regression branch to faster rcnn to predict angle? If so, could someone give me a basic understanding of how to modify the regression branch of the faster rcnn to predict angles of boxes? I assume adding extra regression branch is easier than to create RRPN.</p>
<p>Or to put it simply, how can add oriented bounding box regression to faster rcnn?</p>
Will DDP reduce gradient of only parameters or all tensors?

coincheung: <p>Hi,</p>
<p>When we train the models, in the backward pass, we compute not only graident of model paramters but also tensors. For example, if there is a linear layer:</p>
<pre><code class="lang-auto">y = x * w
</code></pre>
<p>Where <code>w</code> is the model parameter, and <code>x</code> is the input tensor. If we compute backward pass, we should compute not only graident of <code>w</code>, but also graident of <code>x</code>.</p>
<p>What if I train with multi-gpus? I learned from the docs that the gradient of <code>w</code> of each gpu will be averages. But I do not know what would be done with gradient of <code>x</code>. Will graident of <code>x</code> also be averaged by <code>DistributedDataParallel</code> at each layer or only gradient of <code>w</code> get averaged after the backward pass is finished of each iteration?</p>

ptrblck: <p>The parameter gradient should be averaged as the data gradient (in <code>x</code>) depends on the input data.<br>
Since you are training with a different batch on each GPU you wouldn’t need to synchronize the data gradient unless I’m missing something in your use case.</p>

coincheung: <p>Thanks, I figured it out now.</p>
C++ Multiple GPUs for inference

torchy: <p>Hi Everyone,</p>
<p>I am unable to find any documentation on how to set multiple GPUs for inference.</p>
<p>In python the following can be done:</p>
<blockquote>
<pre><code class="lang-auto"></code></pre>
<p>device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)</p>
<p>model = CreateModel()</p>
<p>model= nn.DataParallel(model)<br>
model.to(device)</p>
<pre><code class="lang-auto"></code></pre>
</blockquote>
<p>However for C++ I can’t find the equivalent or any documentation.<br>
torch::nn does not contain either DataParallel or DistributedDataParallel</p>
<p>Any help is appreciated, thanks!</p>

ptrblck: <p><code>libtorch</code> uses the <code>parallel::data_parallel</code> API as seen e.g. <a href="https://github.com/pytorch/pytorch/blob/c371542efc31b1abfe6f388042aa3ab0cef935f2/test/cpp/api/parallel.cpp#L211-L228">here</a>. I don’t know if there are other wrappers, but based on the examples I would guess they should work in a similar way as the <code>DataParallel</code> wrapper in Python.</p>

torchy: <p>Thank you for the snippet!!</p>
<p>I forgot to include</p>
<p><span class="hashtag">#include</span> &lt;torch/nn.h&gt;<br>
<span class="hashtag">#include</span> &lt;torch/nn/parallel/data_parallel.h&gt;</p>

ZhangKangLong: <p>Hi, Sorry to disturb you , I tried the wrapper as you suggest,but the function “parallel::data_parallel” calls another function named “_parallel_run”, which can not be parsed by the compiler of vs2017. It seems like there is no implementation of this function.<br>
Is there any suggestions?</p>

ptrblck: <p>Unfortunately, I don’t know what the current status and support of distributed runs on Windows is so can’t help much <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=10" title=":confused:" class="emoji" alt=":confused:">  Let’s wait for some Windows experts in case they are familiar with it.</p>

ZhangKangLong: <p>Thanks for your reply ~</p>
Torch randn operation gives NaN values in training loop

dome272: <p>Im experiencing something really weird. The torch.randn operation recurrently gives me NaN values in some positions in the tensor. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/7/678ff45989f59303cb49ed98280c5d87b3c39cf2.png" data-download-href="https://discuss.pytorch.org/uploads/default/678ff45989f59303cb49ed98280c5d87b3c39cf2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/7/678ff45989f59303cb49ed98280c5d87b3c39cf2.png" alt="image" data-base62-sha1="eM9BerqUWn8GBiO2qOjhVRn7MfU" width="690" height="362" data-dominant-color="474648"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1202×632 37.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
It gets weirder: the while loop sometimes never gets exited. It seems not to be “random” at this point if NaN values get produced if it once gets caught in it.<br>
And it gets even weirder: When trying the same code, for producing the random latent code, outside in a separate file, <strong>this never happens</strong>. It never creates any NaN values when I run it in this separate file.<br>
I have no idea what to do about this. Did anyone else experience this?<br>
(I also have the whole code on github if that helps: <a href="https://github.com/dome272/ProjectedGAN-pytorch/blob/main/projected_gan.py#L142" rel="noopener nofollow ugc">https://github.com/dome272/ProjectedGAN-pytorch/blob/main/projected_gan.py#L142</a></p>

ptrblck: <p><code>torch.Tensor</code> is not calling <code>randn</code> but is returning an uninitialized tensor, which can contain any values (including invalid values) (same as if you would be using <code>torch.empty</code>).<br>
Since this behavior might not be clear, the usage of <code>torch.Tensor</code> (uppercase T) is deprecated and you should thus use the factory methods via <code>torch.randn</code>, <code>torch.tensor</code> (to initialize the tensor with pre-defined values), <code>torch.empty</code> (in case you want to use uninitialized memory and fill it later) etc.</p>
Inconsistent gather API

jackshi: <p>For 2d torch gather, specifying dim 0 means that for each entry along dim 0, indices are used to gather elements on dim 1. Say dim 0 is the batch dimension, the indices in this case will be used to gather along the features dimension.</p>
<p>For 3d torch gather, specifying dim 0 means that indices are directly applied to dim 0.</p>
<p>The 3d implementation actually generalizes to tensors of all dimensions. Why do we have the 2d gather special case?</p>

ptrblck: <p>Could you post examples for the 2D and 3D case which are showing this inconsistent behavior, please?</p>

jackshi: <p>2D example:<br>
source tensor = [batch, prob_buckets]<br>
idx = [batch, num_samples]</p>
<p>Goal: get probability ratings for each of the idx by gather it from source_tensor.</p>
<p>In 2d case, the approach should be <code>torch.gather(source_tensor, 0, idx)</code>. This is because in order to sample along <code>prob_buckets</code> axies, we need to sample along the 0 dimension. My understanding is that <code>dim=0</code> means that for each entry <code>[prob_buckets]</code> along the <code>batch</code> dimension gets sampled by the <code>idx</code> tensor.</p>
<p>My understanding could be incorrect but if this is the case, then the 3d example with <code>dim=0</code> would sample along the batch dimension for each <code>prob_buckets</code>.</p>

ptrblck: <p>I’m still unsure what exactly is inconsistent.<br>
As described in the <a href="https://pytorch.org/docs/stable/generated/torch.gather.html?highlight=gather#torch.gather">docs</a> the gather operation would be applied as:</p>
<pre><code class="lang-python">out[i][j][k] = input[index[i][j][k]][j][k]  # if dim == 0
out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1
out[i][j][k] = input[i][j][index[i][j][k]]  # if dim == 2
</code></pre>
<p>for a 3D case (and the same logic would still work for the 2D case by removing the <code>k</code> dimension).</p>
<p>Here is a small example:</p>
<pre><code class="lang-python"># 3D
i, j, k = 2, 3, 4
x = torch.randn(i, j, k)
idx = torch.randint(0, i, (i, j, k))

# gather
out = torch.gather(x, dim=0, index=idx)

# manual
out_manual = torch.zeros(i, j, k)
for i_ in range(i):
    for j_ in range(j):
        for k_ in range(k):
            out_manual[i_, j_, k_] = x[idx[i_, j_, k_], j_, k_]

# compare 
print((out==out_manual).all())
# &gt; tensor(True)

# 2D
x = torch.randn(i, j)
idx = torch.randint(0, i, (i, j))

# gather
out = torch.gather(x, dim=0, index=idx)

# manual
out_manual = torch.zeros(i, j)
for i_ in range(i):
    for j_ in range(j):
        out_manual[i_, j_] = x[idx[i_, j_], j_]

# compare 
print((out==out_manual).all())
# &gt; tensor(True)
</code></pre>
I can't find cuda75 for conda install

hitbuyi: <p>the command<br>
conda install pytorch=0.4.1 cuda75 -c pytorch<br>
is provided on our website, but it failed to find cuda7.5, low capacity card should be supported, this thing is meaningful, for a freshman as me, I start my learning by using old laptop, anyone can tell me where to get a copy of cuda75?</p>

ptrblck: <p>I can’t find a CUDA7.5 build, but see CUDA8.0 binaries <a href="https://anaconda.org/pytorch/pytorch/files?version=0.4.1">here</a>.</p>

hitbuyi: <p>thanks a lot, I see it on the website,  and cuda75 for linux, but no cuda75 for windows is available</p>
How autodiff works for matrix

SakuraTea: <p>Hi, I would like to ask a simple question about how autodiff works for vector/matrix. For an instance, if we have C = A.*B where A, B, C are all matrices. When calculating the jacobian matrix of C w.r.t A. does autodiff expand C=A.*B into C_ij= A_ij * B_ij and calculate derivative, or autodiff keeps a rule about this and directly form a result? Thanks in advance for the help.</p>

tom: <p>PyTorch builds a graph of operations to do during the backward pass.</p>
<p>To use a more Pythonic notation, you can do:</p>
<pre><code class="lang-auto">A = torch.randn(5, 5, requires_grad=True)
B = torch.randn(5, 5, requires_grad=True)
C = A @ B
print(C.grad_fn)
</code></pre>
<p>The <code>MmBackward</code> object contains references to A, B (<code>C.grad_fn._saved_self</code> and <code>C.grad_fn._saved_mat2</code>) as well as the instructions how to matrix multiply the Jacobian to a vector in the sense of applying the chain rule (try <code>C.grad_fn(grad_C)</code> with <code>grad_C</code> having the same shape as <code>C</code>).</p>
<p>Best regards</p>
<p>Thomas</p>

SakuraTea: <p>Thank you, Thomas.</p>
<p>Just to confirm, autodiff calculate the gradient of the matrix instead of doing element-wise deviation (e.g. break down Hadamard product and matrix multiplication into element-wise operation and differentiate). Is my understanding correct?</p>

Yaroslav_Bulatov: <p>There’s a tensor-level rule for each core op. If you did per-element differentiation, it would be hard to automatically “reroll” the result back into GPU-efficient operations. Hence when you add a core-level operation working on tensors, you also add a hand-coded derivative for that op that works on tensors (more specifically, vector-jacobian-product).</p>
<p>You can see the issue of doing per-element differentiation by looking at standard derivative results for matrix operations <a href="https://people.maths.ox.ac.uk/gilesm/files/NA-08-01.pdf" rel="noopener nofollow ugc">here</a>. Derivatives of SVD, matrix inverse, and determinant have simple expressions in matrix form, but you can’t easily get this form by doing per-element differentiation.</p>

tom: <p>To add to <a class="mention" href="/u/yaroslav_bulatov">@Yaroslav_Bulatov</a> 's great explanation, the PyTorch derivatives are defined in <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/derivatives.yaml" rel="noopener nofollow ugc">derivatives.yaml</a>. This is what you see in <code>t.grad_fn</code> for tensors computed with gradient requriements.<br>
For functions not in there, the story is a bit more elaborate: these run under autograd and PyTorch records the operations, just like it does for your code. For example <code>einsum</code> will is implemented by calling <code>permute</code>, <code>reshape</code>, and <code>bmm</code>, so these are what you’ll see when looking at <code>.grad_fn</code> or traversing the graph there (by using <code>.grad_fn.next_functions</code> etc. appropriately).</p>
<p>Best regards</p>
<p>Thomas</p>
Cannot modify inputs to MaxPoolGrad

bucky: <p>Hello,</p>
<p>I want to slice the input to the MaxPoolGradNode, but looks like the MaxPoolGradNode should directly get its input(OriginalOutputForResult) from the maxpool node.</p>
<p>line No 182 in <a href="https://github.com/pytorch/glow/blob/master/lib/IR/IRGen.cpp" rel="noopener nofollow ugc">IRGen.cpp</a> cast the input to the MaxPoolGradNode to a MaxPoolNode.</p>
<p>This also creates a problem while partitioning if maxpool node is in a different partition and input to the MaxPoolGradNode becomes a Placeholder.</p>
<p>Here are the lines of code that I was referring to in IRGen.cpp</p>
<blockquote>
<pre><code>auto poolOut = PG-&gt;getOriginalOutputForResult();
</code></pre>
</blockquote>
<pre><code>auto *PI = cast&lt;MaxPoolWithArgmaxInst&gt;(nodeToInstr_[poolOut.getNode()]);
</code></pre>
Is there a way to improve this precision issue?

Jeonghoon_Lee: <p>Hello while playing around with torch.Transforms  (ToTensor &amp; Normalize).<br>
I realized that for some relatively small numbers the output that is  Normalized and then denormalized has ‘-1’ difference.<br>
Confirmed that this happens when I add Normalize transform (No error with just ToTensor). Below is the example code to reproduce. For my application ‘-1’ diff causes a big difference…<br>
Would there be a way to improve this situation, please?</p>
<pre><code class="lang-auto"># Define 'Z' value as numpy
print("Define X ")
X_origin = 16
X = X_origin
X = (np.ones((1,1,3))*X).astype(np.uint8)
print(format(X[0][0][0], '08b'), '\n')
X = Image.fromarray(X)

# To Tensor
print("To Tensor")
transform_list = [transforms.ToTensor()]
t = transforms.Compose(transform_list)
X = t(X)
print(type(X))
print(X.shape)
print(X, '\n')

# Normalize
print("Normalize")
transform_list = [transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))]
t = transforms.Compose(transform_list)
X = t(X)
print(X, '\n')

# DeTensorfy &amp; Denormalize
print("DeTensorfy -&gt; Denormalize")
X = X.cpu().float().numpy()
X = (np.transpose(X, (1, 2, 0)) + 1) / 2.0 * 255.0  
X= X.astype(np.uint8)
print(X, '\n')
print(format(X[0][0][0], '08b'), '\n')

assert X[0][0][0] == X_origin
</code></pre>
<p>X hold 15 istead of 16</p>

Jeonghoon_Lee: <p>Ok I was dumb. Will keep the post and answer, in case someone like me appears…</p>
<p>The issue was with following lines :</p>
<pre><code class="lang-auto">X = (np.transpose(X, (1, 2, 0)) + 1) / 2.0 * 255.0     # 15.9999
X= X.astype(np.uint8)  # Truncates the 0.9999 part..
</code></pre>
<p>I should have rounded X before converting to uint8 dtype</p>
<pre><code class="lang-auto">X = (np.transpose(X, (1, 2, 0)) + 1) / 2.0 * 255.0     # 15.9999
X= np.rint(X)
X= X.astype(np.uint8)  # Truncates the 0.9999 part..
</code></pre>
How to randomly set a variable number of elements in each row of a tensor in PyTorch

ruc: <p>I want to create a zero-one tensor of dimension <code>(n, n)</code>. The ones should be placed randomly, with a cap on the number of ones in each row. Let us say I have a list of length n that has the value of cap for each of the n rows.</p>
<p>One way very close to what I want is to create a random tensor and mask it using <code>torch.topk</code>. However, in <code>torch.topk</code>, <code>k</code> is an integer and remains same for the dimension specified. I need different value of <code>k</code> for every row, so that I can sample variable number of top k elements and turn them into 1s.</p>

Eta_C: <p>Hope could help you.</p>
<pre><code class="lang-python">import torch 

n = 5 
top_k = torch.randint(n, (n,))
print(top_k)

# Step 1, generate a variable number of elements in each row
top_k = torch.nn.functional.one_hot(top_k, num_classes=n)
top_k = 1 - torch.cumsum(top_k, dim=1)  # 
print(top_k)

# Step 2, shuffle each row 
indices = torch.argsort(torch.randn((n, n)), dim=1)
result = top_k[torch.arange(top_k.shape[0])[..., None], indices]
print(result)
</code></pre>
Efficient way to compute cumulative sum of every k consecutive elements of a tensor

elkop: <p>I want to compute cumulative sum of every k consecutive elements of a tensor. More precisely, suppose the tensor is <code>[1,2,3,4,5,6,7,8,9]</code>, then the result should be <code>[1,3,6,4,9,15,7,15,24]</code>. Currently, I am doing this by using <code>torch.cumsum</code> and a for-loop, but I wonder if there is a more efficient way to do this without using a loop.</p>
<p>This is the best solution I have so far. Let me know if anyone comes up with something more efficient.</p>
<pre><code class="lang-auto">k=3
a = [1,2,3,4,5,6,7,8,9]
s = torch.split(a, k)
cumsum = torch.stack(s).cumsum(dim=1)
sum = torch.cat(torch.unbind(cumsum))
</code></pre>

KFrank: <p>Hi Elkop!</p>
<aside class="quote no-group" data-username="elkop" data-post="1" data-topic="136588" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/e/c57346/48.png" class="avatar"> elkop:</div>
<blockquote>
<pre><code class="lang-auto">s = torch.split(a, k)
cumsum = torch.stack(s).cumsum(dim=1)
sum = torch.cat(torch.unbind(cumsum))
</code></pre>
</blockquote>
</aside>
<p>The following is, in essence, the same as your solution, but it might<br>
be a bit more efficient to replace your <code>cat</code> / <code>unbind</code> / <code>stack</code> / <code>split</code><br>
processing with <code>reshape()</code> and <code>flatten()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; k = 3
&gt;&gt;&gt; a = torch.tensor ([1, 2, 3, 4, 5, 6, 7, 8, 9])
&gt;&gt;&gt; sum = a.reshape (k, a.numel() // k).cumsum (dim = 1).flatten()
&gt;&gt;&gt; sum
tensor([ 1,  3,  6,  4,  9, 15,  7, 15, 24])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Eta_C: <p>Convolution with <code>weight = [1, 1, 1]</code> is another way <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Take_along_dim vs gather

Chris_XU: <p>Hi I am wondering what’s the difference between the newly added <code>take_along_dim</code> &amp; <code>gather</code>? It seems that they are pretty much similar other than <code>take_along_dim</code> can have user not specifying <code>dim</code> parameter. In this case seems that torch can find the best broadcast, for example I tried <code>logits</code> of shape (N, K) and use index of <code>logits.argmax(-1)</code> i.e. shape (N,) it can still find the correct broadcast (N, 1), which I believe is not possible by the traditional numpy broadcasting.</p>
<p>Thanks!</p>

ptrblck: <p>This method was adden in <a href="https://github.com/pytorch/pytorch/pull/52833">this PR</a> as part of the numpy compatibility workload and thus might be similar to <code>gather</code>.</p>
RuntimeError: cuda runtime error (8): invalid device function at /opt/conda/conda-bld/pytorch_1503963423183/work/torch/lib/THC/THCTensorCopy.cu:204

Mahsa_Afsharizade: <p>I have a code. I wanted to run it by GPU to accelerate computations. For this purpose I installed NVIDIA driver, CUDA toolkit, and CUDNN.</p>
<p>This is the properties of my system:</p>
<pre><code class="lang-auto">__Python VERSION: 3.5.3 |Anaconda custom (64-bit)| (default, Mar  6 2017, 11:58:13) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
__pyTorch VERSION: 0.2.0_4
__CUDA VERSION: v9.0.176
__CUDNN VERSION: 6021
__Number CUDA Devices: 1
__Devices
index, name, driver_version, memory.total [MiB], memory.used [MiB], memory.free [MiB]
0, GeForce GT 425M, 384.130, 964 MiB, 195 MiB, 769 MiB
Active CUDA Device: GPU 0
Available devices  1
Current cuda device  0
</code></pre>
<p>And this is a part of my code to use GPU:</p>
<pre><code class="lang-auto">    if torch.cuda.device_count() &gt; 1:
        print("Let's use", torch.cuda.device_count(), "GPUs!")
        # dim = 0 [33, xxx] -&gt; [11, ...], [11, ...], [11, ...] on 3 GPUs
        my_rnn_model = nn.DataParallel(my_rnn_model)

    if torch.cuda.is_available():
        print("torch.cuda.is_available() is:",torch.cuda.is_available())
        my_rnn_model.cuda()
</code></pre>
<p>But I received <code>CUDNN-STATUS-ARCH-MISMATCH</code> error.<br>
I googled this error and found that CUDNN needs CUDA compute capability equals or higher than 3.0. But I think that this is 2.1 for me. Therefore I thought that the problem is from my hardware and I can not use GPU.</p>
<p>I decided to come back to my code to run by CPU (without GPU). So I removed that part of my code that if cuda is available then my_rnn_model.cuda().</p>
<p>But Now I expect that the code can be run like before installing CUDA. While not and Now I receive this error:<br>
<code>RuntimeError: cuda runtime error (8): invalid device function at /opt/conda/conda-bld/pytorch_1503963423183/work/torch/lib/THC/THCTensorCopy.cu:204</code></p>
<p>Do you have any suggestion how can I fix it?<br>
Thanks.</p>

hitbuyi: <p>I also encouter this problem, GT 650M,cuda8.0,pytorch0.4.0</p>

ptrblck: <p>I would guess PyTorch <code>0.4.0</code> might use a min. compute capability of 3.5 while your device uses 3.0.<br>
Take a look at the stack trace and try to check which operation / function call fails.<br>
Once you’ve isolated it, check which compute capability introduced it.</p>
Difference between torchvision and detectron2 for Faster R-CNN

gabridego: <p>Hello, I’m trying to reproduce the results of some papers on a public object detection dataset with COCO annotations. They report training a Faster R-CNN with:</p>
<ul>
<li>SGD optimizer</li>
<li>Initial learning rate of 1e-3, reduced to 1e-4 after several iterations</li>
<li>Momentum of 0.9 and weight decay of 5e-4</li>
</ul>
<p>However, if I try to train a torchvision Faster R-CNN pretrained on COCO on the same dataset (following the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">finetuning tutorial</a>) the loss (in particular the rpn_box_reg loss) diverges to NaN after a few iterations, and It only trains when reducing the initial learning rate to 1e-5. Instead, a Faster R-CNN model from detectron2 correctly trains on the same dataset with learning rate 1e-3. In both cases, I am using a ResNet50 backbone with FPN.<br>
I am so wondering if there is any important difference between the two implementations which could lead to that different behavior, and if it suggested preferring one over the other.<br>
The implementations by the paper authors are based on <a href="https://github.com/facebookresearch/maskrcnn-benchmark" rel="noopener nofollow ugc">maskrcnn-benchmark</a> or <a href="https://github.com/jwyang/faster-rcnn.pytorch" rel="noopener nofollow ugc">faster-rcnn.pytorch</a>, which are both deprecated in favor of detectron2.</p>
<p>Thank you in advance for your help.</p>

Miguel_Campos: <p>Hello! Did you get the same APCOCO ? I am comparing both and getting D2 better. Is it you case?</p>
Customized ReLu(at a point other than zero) error

yegane: <p>I am getting an error from this code, but if I use 0 instead of 0.5, it works. I wondered how can it be solved.</p>
<pre><code class="lang-auto">  File "/home/pycy/anaconda3/envs/inpainting_CTSDG/lib/python3.6/site-packages/torch/autograd/__init__.py", line 132, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

</code></pre>
<pre><code class="lang-auto">class MyReLU(torch.autograd.Function):

    @staticmethod
    def forward(ctx, inp):
        ctx.save_for_backward(inp)
        out = torch.zeros_like(inp).cuda()
        out[inp &gt; 0.5] = 1.0
        return out

    @staticmethod
    def backward(ctx, grad_output):
        inp, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[inp == 0.5] = 0
        return grad_input
</code></pre>

ptrblck: <p>Could you post a minimal, executable code snippet which would reproduce this issue, please?</p>

yegane: <p>I wish that I could, but I don’t know how to write a code that can replicate it. I found out that my gradients goes to the roof, and that’s probably the issue. Because when I put 0.9 it works. Anyways, can you check this and see if it does what I intended to do?<br>
I want the activatation layer, pass 1 for values over 0.5 and pass zero for values under 0.5, but in the backward I want it sees a normal ReLu. I know it doens’t make sense, but I need to have values that are either 1 or 0 at the end, and I stell have backpropagation on a ReLu.</p>

ptrblck: <p>The “normal” ReLU backward would be defined as:</p>
<pre><code class="lang-python">    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input &lt; 0] = 0
        return grad_input
</code></pre>
<p>so you might want to change it.</p>

AlphaBetaGamma96: <p>Just a quick comment, could it be an issue that you save <code>inp</code> for backward and then return the output on a different device? So, you have <code>inp</code> on I assume your <code>cpu</code> and <code>out</code> on <code>cuda</code>? And, when computing your loss, <code>grad_output</code> will be on the device of your loss which I assume is <code>cuda</code> and you mask its value by a Tensor which is on the <code>cpu</code>? Surely this would cause some issue?</p>

yegane: <p>Yes, but I don’t want the ReLu exactly, I want the activation function have two behaviour. In the forward, the output would be 1 and 0, but in the backward it sees ReLu backward. So I have a point of discountinoutiy that I want to give its gradiant value in case I had a problem in the backward. Could you please explain to me how can I modify ReLu to a Chimera of Step(forward),ReLu(backward)?</p>

yegane: <p>Thanks, but no it works fine in other values.</p>
How to detach() a rows of a tensor?

Anshul_Paigwar: <p>Is it possible to detach a row or few elements in a tensor?? I tried this</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.rand((5,3),requires_grad = True)
&gt;&gt;&gt; a[2] = a[2].detach()
&gt;&gt;&gt; a[2].requires_grad
True
&gt;&gt;&gt; a.requires_grad
True
</code></pre>

ptrblck: <p>I don’t think it’s possible. Could you just split the tensor and use the parts:</p>
<pre><code class="lang-python">a = torch.randn(5, 3, requires_grad=True)
b = a[2].detach()
a = a[[0, 1, 3, 4]]
b.requires_grad
a.requires_grad
</code></pre>
<p>or could you explain your use case a bit more?</p>

AK47: <p>I found that even though a[2].requires_grad = True, the gradient doesn’t go through a[2]. Maybe this is a bug? More generally, if you want detach() an arbitrary part of a tensor, you can repeat the tensor to 2 copies of it, apply detach() to the second copy and use torch.gather to the repeated tensor as a whole to obtain the desired tensor.</p>

lukas-schott: <p>Strictly speaking this is no “detach” but it this way you can just set them to 0 with hooks:</p>
<pre><code class="lang-auto">import torch

def modify_grad(x, inds):
    x[inds] = 0 
    return x

a = torch.ones((5, 3)).requires_grad_(True)

inds = [2, 4]  # rows which will have zero gradients
b = 2 * a
b.register_hook(lambda x: modify_grad(x, inds))

loss = torch.sum(b**2)
loss.backward()

print('a.grad', a.grad)
</code></pre>
<p>will print</p>
<pre><code class="lang-auto">a.grad tensor([[8., 8., 8.],
        [8., 8., 8.],
        [0., 0., 0.],
        [8., 8., 8.],
        [0., 0., 0.]])
</code></pre>

eschoenfeld: <p>Thanks. This is very helpful!</p>
Grad_fn confusion

semperDM: <p>Hello,</p>
<p>I’m a little confused on what the grad_fn property of a tensor actually means.</p>
<p>I have a tensor that before I take the mean of it has</p>
<pre><code class="lang-auto">grad_fn = &lt;StackBackward&gt;
</code></pre>
<p>after I take the mean of it, this becomes</p>
<pre><code class="lang-auto">grad_fn = &lt;MeanBackward0&gt;
</code></pre>
<p>It’s just not clear to me what this actually means for my network.  The tensor in question is my loss, which immediately afterwards I have</p>
<pre><code class="lang-auto">loss.backward()
</code></pre>
<p>And I’m just looking for clarification on what this changing grad_fn means for my loss function and the backward function here.</p>
<p>Thanks!</p>

ayalaa2: <p>It has to do with the creation of the computation graph. After the mean operation, the tensor is basically just “remembering” the function that created it so that we have a complete history of computation. You’re not gonna want to change <code>grad_fn</code>.</p>
<p><a href="https://pytorch.org/tutorials/beginner/former_torchies/autograd_tutorial.html" rel="noopener nofollow ugc">More information can be found on the AutoGrad page</a></p>

semperDM: <p>So this grad_fn doesn’t change how it’s taking the gradient in the end?  It’s just keeping track of what created the tensor in the first place?</p>

ayalaa2: <p><code>grad_fn</code> is a part of how it computes the gradient in the end. It keeps track of the computation graph so it can perform backpropagation.</p>

semperDM: <p>So it is in fact a bad thing that it’s changing then?  How do I prevent it from changing?</p>

ayalaa2: <p>It’s not a bad thing at all. It’s the process of which gradients are created. You don’t really want to be messing with it. It’s more like you’re getting a new tensor each time.</p>
Should Detectron2 and torchvision to have the same mAP?

Miguel_Campos: <p>Hello community,</p>
<p>I have a custom dataet and using de COCO API to evaluate</p>
<p>1- Faster RCNN with resnet 50 FPN using the official implementation from torchvision<br>
2- Faster RCNN with resnet 50 FPN using the official Detectron2</p>
<p>The results should be similar but i found to D2 like 3-4 points over torchvision.</p>
<p>In both i desactivated all data augmentation and normalized the input size of my images to 1333x800</p>
<p>My question is, what has D2 that make it better than Torchvision in the AP?</p>
<p>PDT: i am not using the “fast coco implementation” of D2 but the official</p>
Gradients are not being updated or stored

Diogo20Mata: <p>So, I’m trying to train a DistilBert model and the gradients are either not being calculated or stored, since, when I print the gradients within a training loop they are always zero. Consequently the weights are not being updated.</p>
<pre><code class="lang-auto">optimizer = optim.Adam(network.parameters(), lr=0.0001)
loss_fn = nn.CrossEntropyLoss().to(device)

train_loader = DataLoader(
    small_train_dataset
    ,batch_size=16
    ,shuffle=True
    #,num_workers=4
)

valid_loader = DataLoader(
    small_val_dataset
    ,batch_size=16
    ,shuffle=True
    #,num_workers=4
)

train_losses = []
val_losses = []

network.train()
for i, batch in enumerate(train_loader): # Get Batch
    input_ids = batch['input_ids'].to(device)
    attention_mask = batch['attention_mask'].to(device)
    labels = batch['labels'].to(device)

    optimizer.zero_grad()
    outputs = network(input_ids, attention_mask=attention_mask) # Pass Batch

    loss = loss_fn(outputs, labels) # Calculate Loss
    loss.backward() # Calculate Gradients
    for name, param in network.named_parameters():
      if (i == 0 or i == 6):
            if param.grad is not None:
                print('GRADIENTS')
                print(param.grad.data)
                print('')
    optimizer.step() # Update Weights
    for name, param in network.named_parameters():
      if (i == 0 or i == 6):
            if 'distilbert.transformer.layer.5.ffn.lin2.weight' in name:
                print('WEIGHTS')
                print(f'BATCH - {i} - TRAIN-- {name}: {param}')
                print('')
      else: continue
    train_losses.append(loss.detach().cpu().numpy())
</code></pre>
<p>Here’s my network</p>
<pre><code class="lang-auto">class Classifier(nn.Module):
    
    def __init__(self, n_classes):
        super(Classifier, self).__init__()
       
        self.distilbert = DistilBertModel.from_pretrained(PRE_TRAINED_MODEL_NAME)
        self.pre_classifier = nn.Linear(self.distilbert.config.dim, self.distilbert.config.dim)
        self.classifier = nn.Linear(self.distilbert.config.dim, n_classes)
        self.dropout = nn.Dropout(0.1)

    def forward(self, input_ids, attention_mask):
        distilbert_output  = self.distilbert(
        input_ids=input_ids,
        attention_mask=attention_mask
        )
        hidden_state = distilbert_output[0]  # (bs, seq_len, dim)
        pooled_output = hidden_state[:, 0]  # (bs, dim)
        pooled_output = self.pre_classifier(pooled_output)  # (bs, dim)
        pooled_output = nn.ReLU()(pooled_output)  # (bs, dim)
        pooled_output = self.dropout(pooled_output)  # (bs, dim) 
        logits = self.classifier(pooled_output)  # (bs, num_labels)

        s_max = F.softmax(logits, dim=1)
        return s_max        
</code></pre>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects raw logits as the model output, so remove the <code>F.softmax</code> in your <code>forward</code> method and check, if this would yield non-zero gradients.</p>

Diogo20Mata: <p>Thank you !! It did yield the non-zero gradients</p>
Multiprogramming and Pytorch Profilers

taehyunzzz: <p>I’m trying to profile the CPU time for dataset preproressing (augmentation) and dataloaders.</p>
<p>Dataloaders are in most environments used in multiprogramming mode, in which many workers in sub-processes fetch input data. Meanwhile torch.autograd.profiler allows profiling for multiprogrammed environments but only keeps track of the the main thread. It does not profile the execution of the child processes do prevent mixed results, according to the documentation of torch profiler.</p>
<p>Is there a way to get the individual profiles of the child data fetching processes? If possible, I would like to find out the average performance of the child processes which will likely be the performance of the whole dataloader.</p>
<p>I think that a mechanism for a child process to return its profile object to the main process upon termination could come in handy. I’ve seen <em>worker_init_fn</em> but no <em>worker_end_fn</em> in torch/utils/data/dataloader.py. Is there anything of the kind? Or should I just get dirty and modify the code by myself?</p>
...stack expects each tensor to be equal size... problem with, supposedly, integer labels

alexeymosc: <p>Hello, I am stuck at preparing a <code>dataset</code> class for multi-class model, that should use MultiClassCross Entropy loss.</p>
<p>Note that<code> data_row.text</code> is a string, and <code>data_row.y#[LABEL_COLUMNS]</code> is an integer, and in my test data ys look like this:</p>
<pre><code class="lang-auto">y
0     7
1    59
2    34
dtype: int64
</code></pre>
<p>Code:</p>
<pre><code class="lang-auto">## Dataset

#del FakeNewsDataset

class FakeNewsDataset(Dataset):
    
    def __init__(
    self,
    data: pd.DataFrame,
    tokenizer: AutoTokenizer,
    max_token_len: int = 2048
    ):
        self.tokenizer = tokenizer
        self.data = data
        self.max_token_len = max_token_len
        
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, index: int):
        data_row = self.data.iloc[index]
        text = data_row.text
        labels = data_row.y#[LABEL_COLUMNS]
        print(labels)
        print(type(labels))
        
        encoding = self.tokenizer.encode_plus(
          text,
          add_special_tokens=True,
          max_length=self.max_token_len,
          return_token_type_ids=False,
          padding="max_length",
          truncation=True,
          return_attention_mask=True,
          return_tensors='pt',
        )
        
        return dict(
          #text=text,
          input_ids=encoding["input_ids"].flatten(),
          attention_mask=encoding["attention_mask"].flatten(),
          labels=torch.LongTensor(labels)
        )

## run sample through the model

fnd = FakeNewsDataset(
  train_df,
  tokenizer
)

sample_batch = next(iter(DataLoader(fnd, batch_size = 20, num_workers = 1)))

</code></pre>
<blockquote>
<p>RuntimeError: stack expects each tensor to be equal size, but got [1] at entry 0 and [2] at entry 2</p>
</blockquote>
<p>This error happens when I change <code>batch_size</code> &gt;= 3 <strong>only</strong>, otherwise it works good.</p>

alexeymosc: <p>So, it always helps to formulate a problem in text.</p>
<p>Fixed it this way:</p>
<pre><code class="lang-auto">labels=torch.LongTensor([labels]).squeeze()
</code></pre>
<p>I think I was forcing the pytorch tensors to generate zero, one, or two random integers (after 0, 1, 2 label codes). My newbe mistakes. And the <code>.squeeze()</code> was later needed to get the right shape for the MultiClass Cross Entropy’s targets.</p>
Can't convert PyTorch model to ONNX: Multiple dynamic inputs needed

Toby_T: <p>I am new to PyTorch, and I built a custom BiLSTM model for sentiment analysis that uses pretrained Word2Vec embeddings. As the text I am passing in is variable in length, my <strong>forward</strong> method needs to pass in both the text and its length to use <strong>pad_packed_sequence</strong>:</p>
<pre><code class="lang-auto"> def forward(self, x, text_len):
</code></pre>
<p>The problem is trying to convert this model to <strong>ONNX</strong>.</p>
<p>My understanding is that the code should be something like this:</p>
<pre><code class="lang-auto">dummy_input_1 = torch.randn(1, seq_length, requires_grad=True).long()
dummy_input_2 = torch.randn(seq_length, requires_grad=True).long()

dynamic_axes = {'input_1' : {1 : 'len'}, 'input_2' : {0 : 'len'}, 
                'output1': {0: 'label'}} 

torch.onnx.export(model,
        args=(dummy_input_1, dummy_input_2),
        f='model.onnx',
        input_names=['input_1', 'input_2'],
        output_names=['output1'],
        dynamic_axes=dynamic_axes)
</code></pre>
<p>However, wouldn’t I need to define <strong>seq_length</strong> first? How do I do this? And is this even the correct approach?</p>
<p>And even if <strong>seq_length</strong> is fixed, I get an error:</p>
<pre><code class="lang-auto">IndexError: index out of range in self

</code></pre>
<p>Any help please?</p>
How to customize activation functions with parameters

kun_yang1: <p>Hi,I’m implementing on a paper. It designs a new activation function called parametric deformable exponential linear units (PDELU).I try to implement such an activation function in pytorch, but I am not very clear about the latitude of the input data（B,C,H,W or C, H, W or H,W）. when pytorch calls the class I defined (PDELU), I perform the forward calculation incorrectly, because the input and weight here are inconsistent.<br>
In addition, I want each input channel to correspond to a parameter of the activation function, but the slicing operation seems to be incorrect. Below, I provide the formula of this activation function and the corresponding gradient expression, as well as my own code. If you can, can you implement a correct version for me? Thank you.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f38ea4bbf40a7d8695e6c3f2166425067efd3c48.png" alt="image" data-base62-sha1="yKBDwUkrS2lD1PGFWjchbK1x9zy" width="660" height="211"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cb27196fcf638e42524a7de5b6f32c7962e68037.png" alt="image" data-base62-sha1="sZaHbl7EBXvkiGfZ8XzgnjCB2n5" width="640" height="98"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88bf5a41561181a06d9ee1bae4b52072f4c8706c.png" alt="image" data-base62-sha1="jvIUejk7EowQx7BMKKmbBRv8stC" width="623" height="443"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/175ca752ea7a5f18e01e0eba1b352a0e4377e11d.png" alt="image" data-base62-sha1="3kFuspu5GJteJxHbvno2tf6NaIl" width="591" height="341"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f08ca5ceee14d986c755c62837e97557986bda9e.png" data-download-href="https://discuss.pytorch.org/uploads/default/f08ca5ceee14d986c755c62837e97557986bda9e" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/0/f08ca5ceee14d986c755c62837e97557986bda9e.png" alt="image" data-base62-sha1="yjZVBOh00To6aCc5AIlhj5zTYeO" width="690" height="410" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">853×507 23.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Question about implmentation difference

jackmsye: <p>I am currently working on the C++ API provided by pytorch. I have some confusion about at::Tensor and torch::Tensor, according to the docs in <a href="https://pytorch.org/cppdocs/" class="inline-onebox" rel="noopener nofollow ugc">PyTorch C++ API — PyTorch master documentation</a> they should be different. However, according to my experiments codes bellow(sorry this might be my first time to post question, i do not know how to highlight codes):</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;torch/torch.h&gt;
#include &lt;type_traits&gt;

int main() {
  at::Tensor a = torch::rand({2, 2}, torch::requires_grad());
  std::cout &lt;&lt; a &lt;&lt; std::endl;
  std::cout &lt;&lt; "is same type: "
            &lt;&lt; std::is_same&lt;torch::Tensor, at::Tensor&gt;::value &lt;&lt; "\n";
  at::Tensor b = torch::rand({2, 2}, torch::requires_grad());
  auto c = a + b;
  c.sum().backward();
  std::cout &lt;&lt; a.grad();
}
</code></pre>
<pre><code class="lang-auto">the output is 
 0.9828  0.3172
 0.7801  0.3766
[ CPUFloatType{2,2} ]
is same type: 1
 1  1
 1  1
[ CPUFloatType{2,2} ]
</code></pre>
<p>it seems at::Tensor is the same as torch::Tensor. I am wondering how the pytorch team achieves this? by namespace aliasing? because they are in different namespace.<br>
Hope someone can answer my questions.</p>
Merge two models

prog_asker: <p>Can I merge two models to extract the feature for each image because those two models ; each one can give me the feature alone … and will the results form concatenating those features will be better than if I used one model of them only</p>

ayalaa2: <p>It is completely valid to combine the features extracted by two different models.</p>
<p>If it’s better is sort of up in the air. It really depends on what the two models are, if they were pretrained, how they were pretrained, and what your downstream task is. It’s really just something you need to experiment with.</p>

prog_asker: <p>Thanks for replying , excuse me how can I start to do that … is there any tutorial for that ?</p>
Torchvision._internally_replaced_utils

JuanFMontesinos: <p>Hi <a class="mention" href="/u/fmassa">@fmassa</a><br>
What’s the plan behind this?<br>
<code>from torchvision.models.utils import load_state_dict_from_url</code><br>
moved to<br>
<code>from torchvision._internally_replaced_utils import load_state_dict_from_url</code></p>
<p>regards!</p>

ptrblck: <p>My guess based on <a href="https://github.com/pytorch/vision/pull/4070">this comment</a> and <a href="https://github.com/pytorch/vision/pull/4072">this PR</a> would be that an internal <code>load_state_dict_from_url</code> method is needed and is thus prefixed with the <code>_</code>.</p>