akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
1top and 5top errors in Pytorch

S_M: <p>Hi all, How can I calculate the k-top error in Pytorch? any help would be appreciated.</p>

Andrei_Cristea: <p>EDIT: This is likely not what you’re looking for. This is a top-k function in PyTorch to get the highest value of a Tensor (including a loss Tensor), but likely you’re looking for the meaning of “top1” and “top5” accuracy like in ImageNet. See my reply below.</p>
<p>Hello, have a look at <a class="mention" href="/u/innovarul">@InnovArul</a> solution to my similar question <a href="https://discuss.pytorch.org/t/ignoring-large-losses-when-backpropagating/148386/2">here</a>.</p>
<p>You would have to slightly adapt it, but it’s the same basic approach outlined under “Ignoring top-k losses”:</p>
<pre><code class="lang-auto">import torch
loss = torch.rand(6, requires_grad=True)
k = 3
sorted_loss, indices = torch.sort(loss)
top_k_losses = sorted_loss[-k:]  # this is how you identify the top losses
</code></pre>
<p>Depending on your specific use case, you can also look at <a href="https://pytorch.org/docs/stable/generated/torch.topk.html#torch-topk" rel="noopener nofollow ugc">torch.topk</a>.</p>

S_M: <p>Hi, thank you for your response.  So, is the top1 error, the largest error value?</p>

Andrei_Cristea: <blockquote>
<p>Hi, thank you for your response. So, is the top1 error, the largest error value?</p>
</blockquote>
<p>Yes, should be.</p>

S_M: <p>What about top5? For it there are 5 values but in the technical report or research paper one value as top5 error is reported. Is it the avarage of the five values?</p>

Andrei_Cristea: <aside class="quote no-group" data-username="S_M" data-post="8" data-topic="153825">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/s_m/48/19733_2.png" class="avatar"> S_M:</div>
<blockquote>
<p>there are 5 values but in the technical report or research paper one value as top5 error is reported. Is it the avarage of the five values?</p>
</blockquote>
</aside>
<p>Hmm, I think your original question was motivated by something different than the highest loss.</p>
<p>Likely what you’re looking for with top1 / top5 is this instead:</p>
<ul>
<li>top 1 means whether the highest probability model prediction matches the label</li>
<li>top 5 means that one of the 5 highest probability predictions match the label</li>
</ul>
<p>See more <a href="https://stats.stackexchange.com/questions/156471/imagenet-what-is-top-1-and-top-5-error-rate" rel="noopener nofollow ugc">here</a>.</p>
SWA(stochastic weight averaging)

Shubham_Singh1: <p>i have few basic doubts about parameters of swa, what is swa_start, swa_frequency, i couldn’t under stand these terms and how to chose values for these parameters? i have one more doubt how to chose swa_lr, do we have to explictly pass it or it can be decided based on lr_scheduler?<br>
thanks!</p>
Huge difference in precision between pytorch and tensorflow

unemployed-denizen: <p>I’m feeding a data into a single convolution layer from Pytorch and TensorFlow. I could make sure that:</p>
<ol>
<li>the input are the same, they are initialized randomly</li>
<li>the kernel weights and bias are same, they are initialized randomly</li>
<li>the way of padding is forced to be the same</li>
</ol>
<p>However, the result shows that there been huge precision differences between the two frameworks. The difference is bigger than 1e-3.</p>
<p>I have attached more details in this <a href="https://github.com/unemployed-denizen/PrecisionProblem/blob/main/Precision%20tf%20vs%20pytorch.ipynb" rel="noopener nofollow ugc">notebook(click here)</a>. This problem is more vital when I tried increasing the input and output channels. While small channel numbers seems to be fine.</p>
<p>I would be really appreciate it someone could tell me what goes wrong with that. Thanks in advance.</p>
<p>Framework Version:</p>
<ol>
<li>Pytorch 1.10.0a0+git593e8f4</li>
<li>TensorFlow 2.5.0 (but it’s tensorflow.compat.v1 in the notebook)</li>
</ol>

ptrblck: <aside class="quote no-group" data-username="unemployed-denizen" data-post="1" data-topic="153780">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/unemployed-denizen/48/50424_2.png" class="avatar"> unemployed-denizen:</div>
<blockquote>
<p>The difference is bigger than 1e-3.</p>
</blockquote>
</aside>
<p>This value doesn’t say much without seeing the ranges or knowing if it’s a relative or absolute error.<br>
Based on your code snippet it seems you are calculating the absolute error in an output range of ~1000 which would be a relative error of <code>~1e-6</code> and expected for <code>float32</code>.</p>

unemployed-denizen: <aside class="quote no-group quote-post-not-found" data-username="unemployed-denizen" data-post="3" data-topic="153780" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/unemployed-denizen/48/50424_2.png" class="avatar"> unemployed-denizen:</div>
<blockquote>
<p>I agree that these errors do not matter, since I have managed to successfully transferred model weights from TensorFlow to Pytorch and test the results today. I was frustrated at first, and thank you for pointing out the relative error.</p>
</blockquote>
</aside>
<p>I agree that these errors do not matter, since I have managed to successfully transferred model weights from TensorFlow to Pytorch and test the results today. I was frustrated at first, and thank you for pointing out the relative error.</p>
Parallel training on multiple GPU

erin: <p>I’m wondering how does the parallel training works (Distributed Data Parallel). I’ve been reading couple of blog posts and here is my understanding, I appreciate if you can correct me if I’m wrong.</p>
<ol>
<li>When the model is copied into multiple GPUs, the weights should all be the same. Is this correct?</li>
<li>After each forward pass, each GPU computes the loss and its gradient individually. Then all of these gradients are aggregated and averaged and passed to the each GPU to update the weights.<br>
Is this correct?</li>
<li>After averaging the gradients and updating the weights, all GPUs should have the same wights. Is this correct?</li>
</ol>
<p>Also, is there any other method that is not doing the average which may cause the models to differ?<br>
Is there any good blog posts that I can read about the detailed theory behind it?</p>

eqy: <p>Yes, that sounds like the correct understanding.</p>
<p>I believe this kind of gradient allreduce-&gt;broadcast with averaging is standard in the data-parallelism regime. However, other methods of utilizing multiple GPUs exist (section 2.1 of a recent paper provides an overview: <a href="https://arxiv.org/pdf/2201.12023.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/2201.12023.pdf</a>), especially when the model activations become large and need to be split across multiple devices.</p>
Is it true the subtraction between two values of loss?

S_M: <p>Hi all, I am going to calculate the difference between two losses as each of them is a mean value of a vector.<br>
loss = loss1 - loss2<br>
the subtraction between them is true?</p>

ptrblck: <p>I don’t fully understand what “true” means in this context. Could you describe your concern a bit more, please?</p>
[Bug Report]Unexpected result torch.abs(x - y) >= 0 while x == y

two_Two: <p>The following code really confuses me , the <code>cond = n &lt; beta</code>  is never true, However when I do backward pass input grad is <code>NaN</code>, I believe this is because <code>0.5 * n**2 / beta</code>, I would like to know how is this possible ? Thanks</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch 
&gt;&gt;&gt; torch.__version__
'1.8.0+cpu'
&gt;&gt;&gt; input = torch.tensor([[-0.6205, 0.225],[-0.8916, 0.4492]], requires_grad = True) 
&gt;&gt;&gt; target = torch.tensor([[-0.6205, 0.225],[-0.8916, 0.4492]]) 
&gt;&gt;&gt; beta = 0 
&gt;&gt;&gt; n = torch.abs(input - target) 
#Since input == target so the cond is never true.
&gt;&gt;&gt; cond = n &lt; beta 
&gt;&gt;&gt; loss = torch.where(cond, 0.5 * n**2 / beta, n - 0.5 * beta)
&gt;&gt;&gt; loss = loss.sum() 
&gt;&gt;&gt; loss.backward() 
&gt;&gt;&gt; input.grad
tensor([[nan, nan],
        [nan, nan]])</code></pre>

eqy: <p>I believe that this is a known effect of the implementation of <code>torch.where</code>: <a href="https://discuss.pytorch.org/t/gradients-of-torch-where/26835" class="inline-onebox">Gradients of torch.where</a></p>
Input form of conv1d

DRISS_ELALAOUI: <p>please, I have an input of the shape</p>
<pre><code class="lang-auto">[1, 2, 20, 135]
</code></pre>
<p>and I have a model</p>
<pre><code class="lang-auto">Net = torch.nn.Conv1d(in_channels=2, out_channels=8, kernel_size=(20, 1), stride=1, bias=True)
</code></pre>
<p>when I execute this line</p>
<pre><code class="lang-auto">Net(input)
</code></pre>
<p>I get the following error:</p>
<pre><code class="lang-auto">RuntimeError: Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [1, 2, 20, 135]
</code></pre>
<p>how I can solve this problem? and thank you</p>

ptrblck: <p><code>nn.Conv1d</code> expects a 3-dimensional input in the shape <code>[batch_size, channels, seq_len]</code> while you are using a 4-dimensional input.<br>
If your input represents <code>[batch_size, channels, height, width]</code> use <code>nn.Conv2d</code> or manipulate the shape to create a 3-dimensional tensor (e.g. by flattening the spatial dimensions into a single one, if this fits your use case).</p>
Converting 4channel images to RGB

sai_m: <p>Hi all,<br>
Is there any simple way to covert 4 channel images into RGB inside transforms? Any help is appreciated.</p>
<p>Thanks</p>

ptrblck: <p>It depends what the 4 channels in the original input represent. E.g. if the 4th channel is the alpha channel and you don’t want to use it, you could transform the image using <code>PIL</code> or just slice the tensor:</p>
<pre><code class="lang-python">transform = transforms.Lambda(lambda x: x[:3])

x = torch.randn(4, 224, 224)
out = transform(x)
</code></pre>
<p>assuming the format is RGBA.</p>
How to DataParallel when batch is a list of Tensors

tanjary21: <p>Hi there,</p>
<p>I’m using the torchvision fasterrcnn pretrained network, which expects a batch input in the form of a list of 3d tensor images and a corresponding list of annotations for each image.</p>
<p>Then, for faster training, I’d like to use DataParallel. But this does not seem to be supported for list input.</p>
<p>Could anyone please suggest a workaround? Thanks in advance</p>
Gradient seems to small when training parameters

S_I_R_U_S: <p>Hi,</p>
<p>I am trying to train a matrix nxn in what looks like a simple scenario. However, I am struggling to obtain a well train matrix when n is big (let’s say above 30). I have the following equation : y = sigmoid(<strong>W</strong>x) where <strong>W</strong> is a nxn matrix, x is the input vector (nx1) and y is the output vector (also nx1). The sigmoid function applies to every elements of <strong>W</strong>x. When, I use a small n, I have no problem to converge back to the original matrix. When I use a bigger n, it’s seems difficult for PyTorch to converge. Here’s my simple class and my code:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):

    def __init__(self, W):
        super(MyModel, self).__init__()
        self.W = W
    
    def forward(self, x):
        return torch.sigmoid(torch.matmul(self.W, x))
</code></pre>
<pre><code class="lang-auto">#Initialise true value

x = np.random.rand(1000, 100)  #Input

W_true = np.random.randn(100, 100)  #True matrix

y = np.zeros((x.shape[0], x.shape[1]))

sigma = lambda x: ((1+np.exp(-x))**(-1))  #sigmoid

for i in range(1000):

    y[i] = sigma(W_true @ x[i])   #True output

# Set up

x = torch.tensor(x, dtype=torch.float32)

W_true = torch.tensor(W_true, dtype=torch.float32)

W = torch.randn(100, 100, dtype=torch.float32, requires_grad=True)

W = nn.Parameter(W)

y = torch.tensor(y, dtype=torch.float32)

learning_rate = 100

model = MyModel(W=W)
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)
loss = nn.MSELoss()

n_iters = 300

for epoch in range(n_iters):
    #Forward pass
    y_pred = torch.zeros(x.shape[0], x.shape[1])
    for i in range(1000):

        y_pred[i] = model(x[i])

    #loss
    l = loss(y, y_pred)

    #Gradient

    l.backward()

    #Update
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>I tried to play with learning rate (which needs to be pretty high which seems weird) and momentum but it does not seems to improve my situation ? Any ideas ? <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>
<p>I am still new to the world of PyTorch (and deep learning) and I would really appreciate if someone could help me in training a matrix in an equation.</p>
<p>Thanks!</p>
Error with at:: cudnn convolution backward input

KQ_P: <p>Hello,</p>
<p>I am setting a new environment from PyTorch 1.6 to PyTorch 1.11.<br>
My major focus is to extend C++ with some ATen functions.<br>
With PyTorch 1.6, I built my custom Conv2d and Linear functions referring to <a href="https://github.com/jordan-g/PyTorch-cuDNN-Convolution" rel="noopener nofollow ugc">https://github.com/jordan-g/PyTorch-cuDNN-Convolution</a> and <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_extension.html</a>.<br>
The first reference was very helpful, from which I used the three functions of <code>at::cudnn_convolution</code>, <code>at::cudnn_convolution_backward_input</code>, and <code>at::cudnn_convolution_backward_weight</code>.</p>
<p>With the PyTorch 1.6, training models with my cpp extension operates well.</p>
<p>However, after migrating to PyTorch 1.11, error occured when calling the last two functions for backwards, i.e., <code>at::cudnn_convolution_backward_input</code> and <code>at::cudnn_convolution_backward_weight</code>; calling the first function, <code>at::cudnn_convolution</code> was fine.<br>
The error message is <code>error: 'cudnn_convolution_backward_input' is not a member of 'at'</code>.</p>
<p>I installed both PyTorch 1.6 and 1.11 via anaconda with the same cuda toolkit of 10.2.</p>
<p>I found that the two backward functions are in the library of PyTorch 1.6 by searching the functions with the command of <code>grep -rnw './' -e 'cudnn_convolution_backward_input'</code> @ <code>.../python3.8/site-packages/torch/</code>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/8/88ceae2dbc02ffbc1892caa360e2782e990c5ebd.png" data-download-href="https://discuss.pytorch.org/uploads/default/88ceae2dbc02ffbc1892caa360e2782e990c5ebd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88ceae2dbc02ffbc1892caa360e2782e990c5ebd_2_690x106.png" alt="image" data-base62-sha1="jwfKib32n4TV0DMHVcWOiNzsIm9" width="690" height="106" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88ceae2dbc02ffbc1892caa360e2782e990c5ebd_2_690x106.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88ceae2dbc02ffbc1892caa360e2782e990c5ebd_2_1035x159.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/8/88ceae2dbc02ffbc1892caa360e2782e990c5ebd_2_1380x212.png 2x" data-dominant-color="1F191A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2970×458 169 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Meanwhile, in the PyTorch 1.11 library, I could not find the two backward functions.</p>
<p>How can I use the functions in the latest version?</p>
<p>Thank you.</p>

eduardo4jesus: <p>Did you get any solution for this?</p>
Apply different encoders to a batch of tensors

Aldebaran: <p>How to split the following <code>batch</code> by <code>modality</code> and forward the splits in different encoders?</p>
<pre><code class="lang-python">
desc_encoder = DescEncoder(...)
img_encoder = ImgEncoder(...)

batch = {

    'idx': torch.tensor([0, 1, 2, 3]),
    'modality': ['desc', 'img', 'desc', 'img'],
    'sample': torch.tensor([
               [   101,  35596, 174818, ...,  0,      0,      0],
               [   101, 322806, 347627,  ..., 0,      0,      0],
               [   101,  35596, 174818,  ..., 0,      0,      0],
               [   101,  35596, 174818,  ..., 0,      0,      0]
               ])
}

desc_sample = # only samples with desc modality
img_sample = # only samples with img modality

desc_rpr = desc_encoder(desc_sample)
img_rpr = img_encoder(img_sample)
</code></pre>

Andrei_Cristea: <p>Hi Celso, would this work?</p>
<pre><code class="lang-auto">mod = np.array(batch["modality"])
sample = batch["sample"]
desc_sample = sample[np.where(mod == "desc")]
img_sample = sample[np.where(mod == "img")]
</code></pre>
Model = model.to(device) problem in the inference using other devices like MYRIAD

gesm: <p>Hi there!</p>
<p>I loaded my model and data to Cuda during the training.<br>
When I run the inference using another device (0) the output is not correct (always 0 values).</p>
<p>Does the Pytorch option to load the model require only the usage of CPU/GPU during the inference?</p>
<p>I want to understand from what the error comes</p>
<p>Thanks</p>
Size not showing correctly

ADONAI_TZEVAOT: <p>I have the following outputs.shape: tensor(-0.0162, device=‘cuda:0’, grad_fn=) torch.Size([])</p>
<p>The following is my error:<br>
ValueError: Target size (torch.Size([1])) must be the same as input size (torch.Size([]))</p>
<p>How do I make ‘outputs’ to be of size 1?</p>
<p>Please help!</p>

ptrblck: <p>You could <code>unsqueeze</code> the scalar (0-dim) tensor:</p>
<pre><code class="lang-python">x = torch.tensor(-0.0162)
print(x)
# tensor(-0.0162)
print(x.size())
# torch.Size([])

y = x.unsqueeze(0)
print(y)
# tensor([-0.0162])
print(y.size())
# torch.Size([1])
</code></pre>
<p>Note that the size is shown correctly and is expected to be empty for a 0-dim tensor.</p>
Convert torch model metts error : ""number of dims don't match in permute

two_Two: <h3>
<a name="system-information-1" class="anchor" href="#system-information-1"></a>System information</h3>
<ul>
<li>OS Platform and Distribution : Linux Ubuntu 18.04</li>
<li>ONNX version : 1.9.0</li>
<li>Python version: 3.8.10</li>
<li>Torch version: 1.8.0</li>
</ul>
<h3>
<a name="describe-the-bug-2" class="anchor" href="#describe-the-bug-2"></a>Describe the bug</h3>
<p>I am trying to convert torch model(.pt) to onnx format but meets unexpected error <code>export failure: number of dims don't match in permute</code>, I inspect into model structure and find out where the problem is and the module is shown below:</p>
<pre><code class="lang-auto">import onnx
import torch 
import torch.nn as nn
import torch.nn.functional as F 
 
class Integral(nn.Module): 

    def __init__(self, reg_max= 16):
        super(Integral, self).__init__()
        self.reg_max = reg_max
        self.register_buffer('project',
                             torch.linspace(0, self.reg_max, self.reg_max + 1))

    def forward(self, x):
        #x.shape (1, 3549, 68)
        x = F.softmax(x.reshape(x.shape[0], -1, self.reg_max + 1), dim = 2)
        #x.shape(1, 14196, 17)
        x = F.linear(x, self.project.type_as(x)).reshape(x.shape[0],-1, 4) 
        return  x

#model setting
model = Integral()
model.to(device) #cpu or gpu 
model.eval()
#input
x = torch.randn(1, 3549, 68).to(device)
f = opt.weights.replace('.pt', '.onnx')  # onnx filename

#opset_version=12
#opt.train =False
torch.onnx.export(model, x, f, verbose=False, opset_version=opt.opset_version, input_names=['images'],
                    training=torch.onnx.TrainingMode.TRAINING if opt.train else torch.onnx.TrainingMode.EVAL,
                    do_constant_folding=True,
                    dynamic_axes= None)
</code></pre>
<p>The code above gives error : <code>RuntimeError: number of dims don't match in permute</code> and I don’t see any wrong with tensor operation in <code>foward</code> . Can anyone help me out with this? Thanks!</p>

two_Two: <p>The problem has been  solved, I change</p>
<pre><code class="lang-auto">self.register_buffer('project'), torch.linspace(0, self.reg_max,self.reg_max + 1) 
</code></pre>
<p>to</p>
<pre><code class="lang-auto">self.project = torch.arange(0, self.reg_max + 1 ).float() 
</code></pre>
<p>and everything works fine.</p>

Thao_Vu: <p>Hi two_Two,<br>
I met the same error. I have changed as you suggest but it doesn’t work. Problem maybe come from F.linear function. Did you change anything else to make that work? Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

Thao_Vu: <p>My problem has been solved, I change</p>
<pre><code class="lang-auto">x = F.linear(x, self.project.type_as(x)).reshape(x.shape[0],-1, 4)
</code></pre>
<p>to</p>
<pre><code class="lang-auto">self.project = torch.t(self.project)
x = torch.matmul(x, self.project).reshape(x.shape[0], -1, 4)
</code></pre>

two_Two: <p>The method I provided was actually wrong, I would try yours once I am free to do this !</p>
Output tensors in PyTorch

Fathima: <p>I am in the process of trying the model. when I visualize I get a zig-zag curve and when I checked the tensor shape I get this . I know it should be an issue with normalization. but how can I fix?</p>
<pre><code class="lang-auto">
print(squeezed_output, target)
tensor([ 31.6040, -13.9625,  30.8022,  -7.8554,  32.3444,   0.2831,  32.4801,
         31.4368,  32.1724, -12.4416,  33.9613,   9.4832,  34.2695,  14.4209,
         35.4740, -21.1821,  35.2900, -23.2053,  34.1040,  -9.1866,  33.0392,
          0.6747,  32.8468, -18.1268,  34.3277,  -5.1512,  32.3085,  29.2341,
         33.6519, -12.5062,  32.1929,  -6.7705], grad_fn=&lt;SqueezeBackward0&gt;) tensor([1., 0., 1., 0., 1., 0., 1., 1., 1., 0., 1., 1., 1., 1., 1., 0., 1., 0.,
        1., 0., 1., 0., 1., 0., 1., 0., 1., 1., 1., 0., 1., 0.])
</code></pre>
<p>but I have already shaped my tensor  when calculating the loss function</p>
<pre><code class="lang-auto">#loss = criterion(squeezed_output,target)
        loss = criterion(squeezed_output.view(-1,1),target.view(-1,1))
        valid_loss = loss.item() * data.size(0)`
</code></pre>
Why there is still a small change in frozen layer parameters?

BierOne: <p>Hi, there. I meet a strange problem in my model. I have frozen the layer ‘fc1’ in my model, but the parameters of such a layer show a small change after the training. The following is my code for the layer-freeze functions, and the way I check the parameters.</p>
<pre><code class="lang-python"># how I freeze the layer
def set_freeze_by_layer_names(model, layer_names, freeze=True):
    if not isinstance(layer_names, Iterable):
        layer_names = [layer_names]
    for name, child in model.module.named_children():
        if name not in layer_names:
            continue
        print("set freeze for layer {} as: {}".format(name, freeze))
        for param in child.parameters():
            param.requires_grad = not freeze

# how I check the parameters
set_freeze_by_layer_names(model,'fc1', freeze=True)
print('before_train', model.module.fc1[0].weight[0, :10])
training(model)
print('after_train', model.module.fc1[0].weight[0, :10])
set_freeze_by_layer_names(model,'fc1', freeze=False)
</code></pre>
<p>The following are the print results:</p>
<pre><code class="lang-auto">set freeze for layer fc1 as: True
before_train tensor([-0.0137,  0.0024,  0.0084, -0.0138, -0.0077, -0.0053, -0.0130,  0.0047,
         0.0057, -0.0151], device='cuda:0')
after_train tensor([-0.0138,  0.0023,  0.0083, -0.0138, -0.0078, -0.0054, -0.0131,  0.0046,
         0.0056, -0.0151], device='cuda:0')
set freeze for layer fc1 as: False
</code></pre>

BierOne: <p>I get the reason! Refer to this <a href="https://stackoverflow.com/questions/53159427/pytorch-freeze-weights-and-update-param-groups" rel="noopener nofollow ugc">link</a>, I also need to set <strong>param.grad=None</strong> if we do not want to accumulate the gradients. So the good freeze function should be:</p>
<pre><code class="lang-auto">def set_freeze_by_layer_names(model, layer_names, freeze=True):
    if not isinstance(layer_names, Iterable):
        layer_names = [layer_names]
    for name, child in model.module.named_children():
        if name not in layer_names:
            continue
        print("set freeze for layer {} as: {}".format(name, freeze))
        for param in child.parameters():
            param.requires_grad = not freeze
            param.grad = None

</code></pre>
Question about inf and nan error

ZiQing_Li: <p>I am not quite clear about the operations that will cause inf and nan? Can you list the operations resulting in the two errors respectively? (e.g. num/0 → nan)</p>

KFrank: <p>Hi ZiQing!</p>
<aside class="quote no-group" data-username="ZiQing_Li" data-post="1" data-topic="153674" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ziqing_li/48/44710_2.png" class="avatar"> ZiQing_Li:</div>
<blockquote>
<p>I am not quite clear about the operations that will cause inf and nan?</p>
</blockquote>
</aside>
<p><code>inf</code> and <code>nan</code> are not really specific to pytorch (or python, for that matter),<br>
but are, rather, part of essentially all modern floating-point arithmetic<br>
implementations (of which almost all essentially follow the IEEE-754<br>
floating-point standard).</p>
<p>For discussions of the <code>inf</code> and <code>nan</code> rules, see, for example, Wikipedia’s<br>
<a href="https://en.wikipedia.org/wiki/Floating-point_arithmetic#Special_values" rel="noopener nofollow ugc">Special values</a> and <a href="https://en.wikipedia.org/wiki/NaN#Operations_generating_NaN" rel="noopener nofollow ugc">Operations generating NaN</a>.</p>
<p>Best.</p>
<p>K. Frank</p>

ZiQing_Li: <p><a class="mention" href="/u/kfrank">@KFrank</a> Quite detailed. Thanks!</p>
Even if the seed is fixed, it is difficult to reproduce the experiment

yipliu: <p>Hi,</p>
<p>I have set seed everything, but the results were very different from experiment to experiment.</p>
<p>How do explain this strange phenomenon?</p>

eqy: <p>In general determinism can be difficult to guarantee as many operations (e.g., if you are using a GPU) are not necessarily deterministic. Have you tried the advice in the reproducibility/randomness post? <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">Reproducibility — PyTorch 1.11.0 documentation</a></p>

yipliu: <aside class="quote no-group" data-username="eqy" data-post="2" data-topic="153681">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eqy/48/46233_2.png" class="avatar"> eqy:</div>
<blockquote>
<p>Have you tried the advice in the reproducibility/randomness post?</p>
</blockquote>
</aside>
<p>Thank you for your advice!</p>
<p>After reading the link you provided, I set the following parameters</p>
<pre data-code-wrap="python"><code class="lang-python">torch.backends.cudnn.deterministic 
</code></pre>
<p>Now, everything is ok</p>
Non FPN ResNet-50 backbone in torchvision

IISCAditayTripathi: <p>I am want to implement Object Detector using torchvision Object detection API. However, I need to have ResNet-50 without FPN as a backbone. How do I use the vanilla ResNet-50 backbone in torchvision?<br>
Kindly help.</p>

Akshay_Gulabrao: <p>I’m trying to do the same thing right now. I think we need to manually construct a backbone object.</p>
Why object detection models taking input as list of tensor?

Mainul: <p>Hello, I was going through the <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html#torchvision-object-detection-finetuning-tutorial" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 1.11.0+cu102 documentation</a>.<br>
There, I see that model (faster_rcnn model with backbone resnet50) takes a list of input image and these images can be various dimentions. But usually these pretrained classifier model’s like resnet or mobilenet does not takes variable size input. For example mobilenet v2 take input image of shaped (3,224,224).  And also, instead of list takes a batch packed in a tensor.</p>
<p>does it doing these processing inside? is there any documentation for that?</p>

ptrblck: <p>Internally <code>GeneralizedRCNNTransform</code> will be used in <code>GeneralizedRCNN</code> and will resize the data using <a href="https://github.com/pytorch/vision/blob/e1f46d42f2acd1039e9a459316bc3daa92d3c983/torchvision/models/detection/transform.py#L123-L133">this code snippet</a>.</p>