akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
(Bug Report) Nvidia Driver Version 536.45 - cuda is not available

coolman: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/8/280da626fb017c45c99f5d9726255732e04a86f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/280da626fb017c45c99f5d9726255732e04a86f0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/8/280da626fb017c45c99f5d9726255732e04a86f0.png" alt="image" data-base62-sha1="5IklbCY2DHe4EpasUSybAyEKQ7u" width="690" height="266" data-dominant-color="232423"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">755×292 7.81 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there anyone facing the same issue like me?<br>
My torch.cuda.is_available() return false after I upgraded my Nvidia Driver to 536.45.</p>
<p>Is there anyway to solve this issue?</p>

ptrblck: <p>What does <code>torch.version.cuda</code> return?</p>

coolman: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a306f9aaccc389b59ca5174ee273beb2459b246.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a306f9aaccc389b59ca5174ee273beb2459b246" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a306f9aaccc389b59ca5174ee273beb2459b246.png" alt="image" data-base62-sha1="f9onTyqWxytx0BaScIYuvzrXZ4y" width="626" height="500" data-dominant-color="222221"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">731×583 18.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Backpropagation with iterative variable update

Mehdi_Ibrahimli: <p>Hi everyone, I have a simple training loop</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):                   
    for i,data in enumerate(training_set):       
        optimizer.zero_grad()                             
        output_layer = PIAE_model(data)
        loss = criterion(output_layer,data)
        loss.backward()                            
        optimizer.step()                          
        loss_log.append(loss.item())
</code></pre>
<p>model methods look like this</p>
<pre><code class="lang-auto">   def inverter(self,x):
        x = self.up(self.act(self.bn1(self.conv1(x))))     # convolution, batch normalization, activation, upsampling
        x = self.up(self.act(self.bn2(self.conv2(x))))    
        x = self.up((self.bn3(torch.abs(self.conv3(x)))))   
        return x

   def findif(self,vel_model):
       for n in range(0,self.nt):
           self.p[1] = self.p[1]+self.q[n]               
           self.p = vel_model * self.p.detach()
           self.traces[n] = self.p[1]
        return self.traces
    
    def forward(self, x):
        vel_model = self.inverter(x)
        seis_model = self.findif(vel_model)
        return seis_model
</code></pre>
<p>Where q and p are torch tensors with require_grad(True)<br>
Training like this gives NaN values in the loss and setting detect anomaly to True raises:<br>
<em>Function ‘CudnnBatchNormBackward0’ returned nan values in its 0th output</em></p>
<p>Updating the variable <code>self.p</code> interferes with the backpropagation. And I am sure that the error is here  <code>self.p[1] = self.p[1]+self.q[n]</code> without it training works just fine.</p>
<p>This is simplified but relevant version of my code, I can of course provide the full code but it is rather convoluted with mathematical operations in loops in findif method.</p>
<p>Please help me rewrite this line so the gradients are calculated properly.</p>

Mehdi_Ibrahimli: <p>Solved it</p>
<pre><code class="lang-auto">test = []
for i in range(0,self.nx): 
     test.append(self.p[i].detach())                
test[1] = test[1]+ self.q[n]
self.p = torch.stack(test)
</code></pre>
<p>Would love to see a prettier solution tho.</p>
Load_state_dict padding and resizing pre trained model

wgeul: <p>Hi,</p>
<p>I’d like to ask advise on a functionality I would expect from <code>load_state_dict</code>.</p>
<p>The current implementation will fail on a size mismatch or key mismatch. Specifying <code>strict=False</code> will ignore and skip such cases.</p>
<p>My desired functionality is to address the size mismatch in a way that allows the user to benefit from pre-trained tensors, despite a structural change in the network.</p>
<p>Concretely:</p>
<ul>
<li>in case the new tensor is larger, randomly initialize (or zero init) the residual.</li>
<li>in case the new tensor is smaller, retain the overlapping shape and throw away the rest.</li>
</ul>
<p>Is it possible to do this?</p>
<p>I often find myself with this need: I pre-train a smaller and less deep model - plateau in terms of loss - subsequently I’d like to carry this experience towards the next model generation: a broader and/or deeper model - saving me some training time.</p>

ptrblck: <p>Yes, this should be possible my directly manipulating the stored parameters and buffers in the <code>state_dict</code> using your padding/cropping technique. You would then need to experiment with these techniques to see how much worse the overall training would be and if a training from scratch would yield the same results.</p>
Does pytorch 2.0.1 support cuda version 12.2?

karthik_rathod: <p>For some reason am not able to access my gpu using torch  think it might be dues to cuda 12.2.</p>

ptrblck: <p>Why do you think it would be due to CUDA 12.2?<br>
The PyTorch binaries ship with their own dependency and your locally CUDA toolkit will be used if you build PyTorch from source or custom CUDA extensions.<br>
You would only need to properly install an NVIDIA driver.</p>
Multistream network with lateral connection in pytorch

Neskelogth: <p>I need to work on a project for which I need two different branches for a neural network using PyTorch, fusing the output at a late stage. I would like to implement lateral connections so that the networks are able to share information, but I couldn’t find any documentation about it. Is it possible to implement these types of connections in PyTorch in general? And if the two branches are a Convolution branch and a GNN/GCN branch?</p>
When plotting the confusion matrix, how to reorder the class according to the super class

nikiguo93: <p>Hi, I am now plotting the confusion matrix. In my case, the model was trained to predict 1654 categories. then the confusion matrix is of size 1654 * 1654. But these categories can be grouped into 49 super classes. Then I intend to reorder the confusion matrix and make the categories belonging to the same super class together. Does anybody know how to realize it?</p>
How to get intermediate results of ops in torchdynamo?

weilinquan: <p>I use AOTAutograd to produce FX graphs consisting of canonical Aten ops. It likes this.</p>
<pre><code class="lang-auto">%arange : [num_users=1] = call_function[target=torch.ops.aten.arange.start](args = (0, 13), kwargs = {dtype: torch.int64, device: cpu, pin_memory: False})
    %unsqueeze : [num_users=1] = call_function[target=torch.ops.aten.unsqueeze.default](args = (%arange, 0), kwargs = {})
    %view : [num_users=64] = call_function[target=torch.ops.aten.view.default](args = (%unsqueeze, [-1, 13]), kwargs = {})
...
</code></pre>
<p>For some reasons, I want to get intermediate results in these ops, such as the value of %unsqueeze. How can I get it? Thank you for your help.</p>
How to preprocess data for nn.Embedding correctly?

loveis98: <p>I’m quite new to using Pytorch; the question is how to preprocess data for nn.Embedding correctly?<br>
I see it takes only integers, but should these integers be like a sequence (0, 3, 4, 1, 2, 3, …) whose unique values start with 0, or any integer values are allowed (45, 98, 120, 45, …) as long as the number of unique values matches the input dimension?</p>

ptrblck: <p>The input should contain indices in the range <code>[0, num_embeddings-1]</code>.</p>
Changing values of model's parameters after wrapped with DDP

AlanSouthwark: <p>In DDP docs, there is a note that warns the user should never try to change the model’s parameters</p>
<blockquote>
<p>You should never try to change your model’s parameters after wrapping up your model with <code>DistributedDataParallel</code> . Because, when wrapping up your model with <code>DistributedDataParallel</code> , the constructor of <code>DistributedDataParallel</code> will register the additional gradient reduction functions on all the parameters of the model itself at the time of construction. If you change the model’s parameters afterwards, gradient reduction functions no longer match the correct set of parameters.</p>
</blockquote>
<p>I wonder if it is allowed to just change the values of the model parameters after wrapping the model with DDP.</p>

ptrblck: <p>Changing the values would create diverged parameter sets between the ranks unless you guarantee that all ranks apply exactly the same manipulation. I would still be careful in making sure not to replace the parameters.</p>
How does F.interpolate work for downsampling?

ymin2570: <p>I have a question about F.interpolate, same as torch.nn.functional.interpolate.</p>
<p>when we are performing downsampling using F.interpolate(tensor, size, mode=‘bilinear’, align_corners=False), how does it working?<br>
Is it performing average pooling or max pooling?</p>
<p>And is anti-aliasing necessary? aliasing will be occurred?<br>
Additionally, what’s the method for applying anti-aliasing? Is Low Pass Filtering with Gaussian kernel?</p>

ptrblck: <p>The <a href="https://en.wikipedia.org/wiki/Linear_interpolation">Linear Interpolation Wikipedia article</a> might be a good starter which explains how values “between indices” are selected and how the target value is then calculated using the linear interpolation.<br>
Anti-aliasing can occur during downsampling and you can use a low-pass filter.</p>
Why am I not able to calculate the gradient inspite of setting the requires_grad=True

Bluegirl123: <p>Hi I am trying to create a diffractive neural network model using pytorch. Although I set the requires_grad=True for the parameters that I want to calculate the gradients, I am unable to get the gradients. It shows the gradients as none type. So I am unable to update the parameters. Can anyone please suggest me the possible reason for it?</p>

ptrblck: <p>Could you post a minimal and executable code snippet to reproduce the issue, please?</p>
I want to create a Bert model which can predict the SentiwordNet Avg score (pos_score - neg_score) for every word in the sentence. The score to predict is between 0-1

Singh_s_EdiT: <p>dict_keys([‘review_text’, ‘input_ids’, ‘attention_mask’, ‘targets’, ‘SentiwordAvgScore’])<br>
sentence_length = 30<br>
batch_size = 32<br>
review_text = torch.Size([32, 30])<br>
input_ids =torch.Size([32, 30])<br>
attention_mask = torch.Size([32])<br>
SentiwordAvgScore =torch.Size([32, 30])</p>
<p>If the sentence has 30 words then the there will be 30 different values between 0-1 to be predicted. I have implemented the Bert model.<br>
How can I build a neural model output layer and which loss function can i use?</p>
Correct return for undefined gradients?

randolf-scholz: <p>I am writing custom C++ autograd functions, and found several different declarations for undefined gradients in <code>torch::autograd::variable_list</code> in the pytorch source code. (I.e. for non-differentiable inputs)</p>
<ol>
<li><code>torch::Tensor()</code> in <code>custom_operator/op.cpp</code></li>
<li><code>undef</code>, defined as <code>Tensor undef;</code>, in <code>sparse_ops_gpu.cpp</code></li>
<li><code>torch::autograd::Variable()</code> in <code>empedding_backward_dense_host.cpp</code></li>
</ol>
<p>My questions:</p>
<ol>
<li>What is the difference between ① and ②? Does ② skip initialization of the tensor</li>
<li>Which of ①-③ is preferable?</li>
<li>Would it make sense to do <code>static const Tensor undef;</code> instead of just <code>Tensor undef;</code> in ②? (I am new to C++)</li>
</ol>
How can I calculate the grad in func jacrev()

xuelanghanbao: <p>Hi, everyone. Here is a function to calculate the Neural Tangent Kernel (NTK)</p>
<pre><code class="lang-auto">def ntk(model, x):
    from torch.func import functional_call, vmap, jacrev

    params = dict(model.named_parameters())

    def fnet_single_torch(params, x):
        y = functional_call(model, params, x.unsqueeze(0)).squeeze(0)
        return y

    jac1 = vmap(jacrev(fnet_single_torch), (None, 0))(params, x)
    jac1 = jac1.values()
    jac1 = [j.flatten(2) for j in jac1]

    jac2 = vmap(jacrev(fnet_single_torch), (None, 0))(params, x)
    jac2 = jac2.values()
    jac2 = [j.flatten(2) for j in jac2]

    # Compute J(x1) @ J(x2).T
    einsum_expr = None
    if compute == "full":
        einsum_expr = "Naf,Mbf-&gt;NMab"
    elif compute == "trace":
        einsum_expr = "Naf,Maf-&gt;NM"
    elif compute == "diagonal":
        einsum_expr = "Naf,Maf-&gt;NMa"
    else:
        assert False

    result = torch.stack(
        [torch.einsum(einsum_expr, j1, j2) for j1, j2 in zip(jac1, jac2)]
    )
    result = result.sum(0)
    return result.squeeze()
</code></pre>
<p>Now, I need to add a transformation to the network output and calculate the gradient term in this transformation. How can I modify this code?</p>
<p>I have tried adding transformations directly to <code>fnet_single_torch</code> function like below, but I noticed that when using function <code>jacrev()</code>, the incoming <code>x</code> will not require grad</p>
<pre><code class="lang-auto">def output_transform(x,y):
        return autograd.grad(y, x, grad_outputs=torch.ones_like(y))[0]

def fnet_single_torch(params, x):
        y = functional_call(model, params, x.unsqueeze(0)).squeeze(0)
        y = output_transform(x,y)
        return y
</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/xuelanghanbao">@xuelanghanbao</a>,</p>
<p>If you want a function that computes the derivatives of your model output with respect to the inputs within the <code>torch.func</code> namespace, you can just use the <code>torch.func.grad</code> function,</p>
<pre><code class="lang-auto">def f(params, x);
  return functional_call(model, params, x.unsqueeze(0)).squeeze(0)

grad_f = torch.func.grad(f, argnums=(0))(params, x)
</code></pre>
<p>If your input has a batch dimension you can compose the gradient within a vmap, because otherwise you’ll compute the gradient across different samples in your batch (which isn’t what you want),</p>
<pre><code class="lang-auto">grad_f = torch.func.vmap(torch.func.grad(f, argnums=(0)), in_dims=(None,0))(params, x)
</code></pre>
Conversion of .pt to .ptl for classification models giving very poor results

madhur04: <p>I am trying to build a object detection model which detects flies on cow. I have done a good job detecting flies using yolo frameworks but this only happens when the quality of the image is top notch as well as the position of the cow is closer to the screen. i.e If the image is taken from a short distance. So, I planned to reject the bad images ( quality as well as position ) using a classification model. I build a 5 class classification model -</p>
<p>bad class - having blurry images and cow which are far away.<br>
difficult - some what decent images as compared to bad<br>
good - good quality and short distance cows.<br>
no_cow - images in which cows are absent. ( Basically the dataset consists of green pasture images without any cow in them )<br>
not_cow - Animals other than cow.</p>
<p>I trained the model in pytorch and the <strong>.pt</strong> is doing a good job in predicting the classes with a validation accuracy of 77%. But when I convert the <strong>.pt</strong> file to <strong>.ptl</strong>  and run inference on mobile using same set of images ( I am using react-native-pytorch-core library ) I get a validation accuracy of 62%. But the .ptl model 99% of the time is only predicting among the good,no_cow and not_cow images whereas the .pt model is somewhat equally distributed among all the classes. What may me the reason?</p>

madhur04: <p>Okay, so after trying various methods we figured out above issues to some extent. Writing this as it may help somebody someday.</p>
<p>As we were using a pre-trained ImageNet model , normalizing the tensor with values MEAN = [0.485, 0.456, 0.406] and STD = [0.229, 0.224, 0.225] instead of simply passing [0.5,0.5,0.5] solved few issues. Next we compared the image tensors which were getting passed as input to the model and we found out that they were quiet different. We were using pytorch in the web code and react-native-pytorch-core library for our mobile code. Visualizing the image tensors helped us a lot. As per the documentation of react-native-pytorch-core and the example provided by them for image classification, we had used <strong>centerCrop</strong> function due to which the image was getting cropped and hence the results varied significantly between web and mobile. Removing the centerCrop step and resizing the image to the desired size reduced the difference between web and mobile results to a great extent. Some difference is also attributable to the differences in underlying resizing functions in opencv and PIL.</p>
How to know the type of a tensor?

ADONAI_TZEVAOT: <p>I tried print(type(a_tensor)) but its only telling me that its a tensor. How do i know if it is float type or double type or whatever it may be ? Please help</p>

khushi-411: <p>Hi,<br>
You can find data types of tensors using:</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.tensor([1, 2, 3])
&gt;&gt;&gt; a.dtype
torch.int64
</code></pre>
<p>Hope it helps. Thanks!</p>
What's recommend to serialize a compiled torch model

Yuzhui: <p>Hi guys,</p>
<p>We are testing out pytorch 2.0 internally and it seems that we cannot export a pytorch compiled model into torchscript using .trace() or .script()</p>
<p>Has anyone ever tried to serve a compiled torch model?</p>
<p>Thanks,<br>
Yuzhui</p>

marksaroufim: <p>So question why do you need to export it are you deploying to a mobile runtime?</p>
<p>If not what’s wrong with just using python in prod? And if that’s not OK you can take a look at <code>torch.export.export()</code> in the nightlies</p>

Mah_Neh: <p>As you probably know the web browser won’t run python code. So if your model runs in the backend you can just use python, unless the server is also in NodeJS in which case you have to convert it to ONNX.</p>
<p>If instead the model runs in the client  i.e web application, there is no other way but to turn it into ONNX.</p>
<p>The steps are:</p>
<ol>
<li>Export using <code>torch.export.onnx(...)</code></li>
<li>Use ONNX runtime which is a javascript package in your project to load and run the model (you need to implement pre + post processing)</li>
</ol>
<p>You can also further optimize the inference by using onnxruntime python package to conver the onnx model to ORT. This is secondary though.</p>
<ul>
<li>Both ONNX and ORT are supported by <code>onnxruntime-web</code> which is in the npm repository.</li>
</ul>
<p>You may face some errors related to WASM, in which case you will find how to fix it by Googling or asking.</p>
<p><a class="mention" href="/u/yuzhui">@Yuzhui</a></p>
Visualising how my CNNs is doing?

Mah_Neh: <p>I’m track both the <strong>loss</strong> and the <strong>accuracy</strong> of both training and test sets at training time.</p>
<p>However, I want to have a deeper understanding of my neural network.</p>
<ul>
<li>What is the best way to inspect what the inner layers are computing ?</li>
</ul>
<p>For example, they do it in the animation from Stanford CS site: <a href="http://cs231n.stanford.edu/" rel="noopener nofollow ugc">http://cs231n.stanford.edu/</a></p>
<p>I’d like to do the same with my Torch based CNN.</p>
<ul>
<li>Can you suggest any starting point ?</li>
</ul>
<hr>
<p>I’m also trying to start tracking the gradients in training time, and would welcome any pointer as to how could I track this in a useful, and simple way (I am rather a beginner.)</p>
Can we use int8 activation quantization in pytorch

Chenpeng_Z: <p>when I specify dtype for activation as qint8, quantization is not applied as expected.<br>
here is script:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, padding=1)
        self.bn1 = nn.BatchNorm2d(64)
        self.relu1 = nn.ReLU(inplace=True)
        self.conv2 = nn.Conv2d(64, 64, kernel_size=3, padding=1)
        self.bn2 = nn.BatchNorm2d(64)
        self.relu2 = nn.LeakyReLU()

    def forward(self, x):
        x = self.conv1(x)
        x = self.bn1(x)
        y = self.relu1(x)
        x = self.conv2(y)
        x = self.bn2(x)
        x = self.relu2(x)
        return x


example_inputs = torch.randn(1, 3, 224, 224)
float_model = MyModel()
float_model.eval()
qconfig_sub = torch.ao.quantization.QConfig(
  activation=observer.MovingAverageMinMaxObserver.with_args(qscheme=torch.per_tensor_symmetric, dtype=torch.qint8),
  weight=observer.MovingAveragePerChannelMinMaxObserver.with_args(qscheme=torch.per_channel_affine, dtype=torch.qint8,)
)
qconfig_mapping = QConfigMapping()
qconfig_mapping.module_name_qconfigs = {"conv1": qconfig_sub, "relu2": qconfig_sub}
# print(qconfig_mapping.to_dict())
prepared_model = prepare_qat_fx(float_model, qconfig_mapping, [example_inputs], ) 
quantized_model = convert_fx(prepared_model)
quantized_model(example_inputs)
</code></pre>
<p>graph:</p>
<pre><code class="lang-auto"># To see more debug info, please use `graph_module.print_readable()`
graph():
    %x : [#users=1] = placeholder[target=x]
    %conv1 : [#users=1] = call_module[target=conv1](args = (%x,), kwargs = {})
    %conv2 : [#users=1] = call_module[target=conv2](args = (%conv1,), kwargs = {})
    %relu2 : [#users=1] = call_module[target=relu2](args = (%conv2,), kwargs = {})
    return relu2
</code></pre>
<ol>
<li>I assume we have to specify as uint8, but for Sigmoid, LeakyReLU, does uint8 lead to accuracy drop?</li>
<li>if so, how could I extend supports for int8 activation quantization<br>
appreciate any feedback and ideas, thanks!</li>
</ol>

111357: <p>The quantization is not applied due to the backend not supporting act(uint8) + weight(int8), you can find more details about the backend config <a href="https://github.com/pytorch/pytorch/blob/c99895ca6f98eab834611e007de772534fd57fb9/torch/ao/quantization/backend_config/x86.py#L27-L32" rel="noopener nofollow ugc">here</a>.</p>
<p>To extend the support for int8 activation, I think you need to customize the kernel that can handle act(uint8) + weight(int8).</p>

Chenpeng_Z: <p>thanks, can you explain <code>customize the kernel</code> with more details?</p>

rick1: <p>same problem, can you explain <code>customize the kernel</code> with more details?</p>
Checkpointing with lr_scheduler

kushaangupta: <p>Hi, I’m using checkpointing to resume training of a model. However, I am also using <code>torch.optim.lr_scheduler.OneCycleLR</code> alongside which is giving me some problems. I initially train a model for 10 epochs and saved the best checkpoint and then set <code>max_epochs</code> to 20 to continue training for 10 more epochs but I get the following error from lr_scheduler. Seems like when I load the checkpoint the lr_scheduler is still stuck on initial max limit on number of steps set by steps_per_epoch argument in OneCycleLR.</p>
<pre><code class="lang-auto">ValueError: Tried to step 101 times. The specified number of total steps is 100
</code></pre>
<p>I have tried setting the <code>steps_per_epoch</code> arg of OneCycleLR or the <code>total_steps=self.trainer.estimated_stepping_batches</code> but nothing helps.</p>

Mah_Neh: <p>Can you show how you configure the optimiser ? Did you try with verbose=True to get mode information ?</p>

kushaangupta: <p>Sure, here goes:</p>
<pre><code class="lang-python">optimizer = torch.optim.AdamW(
    self.parameters(), weight_decay=1e-2)
scheduler = torch.optim.lr_scheduler.OneCycleLR(
    optimizer, max_lr=3e-2,
    epochs=10, steps_per_epoch=len(train_dataloader)
)
</code></pre>
<p>Using <code>verbose=True</code> in scheduler gives me log of what the learning rate is at each step and a stack trace on error but I do not know what to do with it.</p>
<p>I was hoping to find some standard practice to update the steps property of scheduler after loading the checkpoint but couldn’t find it. Another hack I was thinking was to not use the scheduler if I’m loading from a checkpoint, which does work but it all seems too much. I hope there’s a better way.</p>
Does torch.load or torch.jit.load need nvidia drivers?

thingsofleon: <p>does torch.load or torch.jit.load need nvidia drivers?</p>
<p>the model was created using CUDA, however, we are trying to run inference on a cpu only machine.</p>
<p>everything is set to use the cpu in the inference code, however we are getting a runtime error for no nvidia drivers on the torch.load call.</p>
<p>Any way to get around this?</p>

ptrblck: <p>Are you already using the <code>map_location</code> argument to load the data to the host? If not the load command would try to load the data to the device it was stored from, which might be the GPU.</p>

thingsofleon: <p>Ya. we tried many torch.load(…, map_location=…) that we have seen on here. Get a Runtime Error for no Nvidia Drivers for all we tried.</p>
<p>The model was saved to disk via a GPU train, and we are trying to load it on a CPU only machine.</p>
What will be the correct world_size in the training with multiple nodes?

devsy: <p>Hi,<br>
I was running DDP tutorial on two machines.  (ddp tutorial- <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.0.1+cu117 documentation</a>)</p>
<p>I used torchrun, with following command<br>
<code>torchrun --nnodes=2 --nproc_per_node=$NUM_GPUS --rdzv_id=100 --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:29400 ddp_test_elastic.py</code></p>
<p>when I printed dist.world_size() on node 1, it says 4  but on node 2, it says 6  (node_rank=0 has 4 gpus and node_rank=1 has 2 gpus in my setup)</p>
<p>Is this right?<br>
I think it is correct for the second node, which prints 6, but maybe wrong for the first node.<br>
If the first node is aware of the second node, then shouldn’t it be 6?</p>
Cannot import name 'QuantStub' from 'torch.ao.quantization'

AliceKoh: <p>Hi I am new to pytorch.</p>
<p>I encounter an issue when I run the below. Can someone assist ?<br>
from torchvision import datasets, transforms</p>
<p>Error message:<br>
ImportError: cannot import name ‘QuantStub’ from ‘torch.ao.quantization’</p>

ptrblck: <p>Are you seeing this error while just running <code>from torchvision import datasets, transforms</code> or any other imports?<br>
Could you post the versions reported by <code>pip list | grep torch</code> and/or <code>conda list | grep torch</code>?</p>

AliceKoh: <p>Hi ptrblck,<br>
Thanks for reply. Someone suggested that I take away the .ao from all the error message files. So i just took away .ao from the below files and now there is no error message. Hope it really resolve the issue.</p>
<p><strong>init</strong><br>
mobilenetv2<br>
mobilenetv3</p>

jerryzh168: <p>you probably have an older version of pytorch? we are deprecating the torch.quantization namespace, and torch.ao.quantization is the recommended one</p>

AliceKoh: <p>Hi Jerry,<br>
Oh I see. May I know what version of pytorch &amp; python should I install so that I can run pytorch smoothly?</p>

Luis_Ivan_Umpire_Alv: <p>Updating pytorch fixes the problem:</p>
<p>pip install --upgrade torch torchvision</p>
When will the callback be triggered in PyTorch DDP bucket allreduce?

zhuce_yong: <p>Hello,<br>
Can someone help me understand when the callback function is called in DDP reducer bucket allreduce? Previously I thought it’s called when the CUDA stream on GPU is done, but now I think it’s called when the work on CPU is done.<br>
The callback fundtion is defined in torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:collective method.</p>
<pre><code class="lang-auto">    if (work-&gt;recordFunctionEndCallback_) {
      work-&gt;future_-&gt;addCallback([work](at::ivalue::Future&amp; /* unused */) {
        work-&gt;recordFunctionEndCallback_();
      });
    }
</code></pre>
Training Stops: CPU stays busy, GPU goes to zero

Jhall: <p>Im training an RL agent on another vision task with the simple neural net from one of the first pong papers. Everything goes smoothly for 48 epochs, each having 240 steps. 7950x CPU utilization at about 18%, RTX 3090 GPU utilization at about 8% and using about 2 gigs of memory. Temps low on both. RAM at 44 of 64 gigs. Once I reach 48 epochs CPU keeps being utilized, GPU memory stays the same, GPU utilization goes to zero and no new epochs complete. I restart the program several times, always getting to epoch 48, and then the same problem. I restart my computer and now I get to epoch 51, and then the same problem. Restart the program and epoch 51 again before the same problem. Any advice on where to begin trouble shooting this?</p>
Illegal memory access error in torch.max_pool2d

mandreu: <p>Hi,</p>
<p>I am getting the following error, with <code>torch.backends.cudnn.enabled = False</code> set:</p>
<pre><code class="lang-auto">  File "/opt/conda/envs/_/lib/python3.9/site-packages/torch/_jit_internal.py", line 422, in fn
    return if_false(*args, **kwargs)
  File "/opt/conda/envs/_/lib/python3.9/site-packages/torch/nn/functional.py", line 720, in _max_pool2d
    return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: CUDA error: an illegal memory access was encountered
</code></pre>
<p>Without <code>torch.backends.cudnn.enabled = False</code>, this is the error I get:</p>
<pre><code class="lang-auto">  File "/opt/conda/envs/_/lib/python3.9/site-packages/torch/nn/functional.py", line 2283, in batch_norm
    return torch.batch_norm(
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.
</code></pre>
<p>As far as I can tell, the error occurs when <code>input</code> to <code>torch.max_pool2d</code> exceeds a certain size (e.g. while [16, 64, 1200, 1712] still works, [16, 64, 1184, 1824] already throws the error). I can work around this by reducing the size per image or by reducing the batch size (the first element, i.e., 16).</p>
<p>It is reliably reproducible: could not pinpoint the exact threshold as input size varies due to augmentations, but when I go high enough with the initial image size, it always occurs, at the first batch.</p>
<p>From that I assume it is an OOM issue – but not sure if that is correct or if there is a bug, there are any configs or something to be changed? The thing is that I also get regular OOM errors in other places so it confuses me that that one differs. Either it only differs in error handling or it differs in the actual error; I can’t tell.<br>
Can this type of error actually point to an OOM issue or is it a different issue?</p>

ptrblck: <p>Could you post a minimal and executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>?</p>
Training models inside hooks

okm02: <p>I would like to train some linear layers on top of different intermediate layers in my network. Is this possible to do with hooks ? i.e : initialize my backbone feature extractor and freeze it; Initialize multiple linear layers and run the forward and backward of these models in nn.hooks ?</p>

ptrblck: <p>Yes, forward hooks allow you to use the forward activation, which could then be fed to another model for training.</p>
Finetuning a "unfrozen" model does not preserve accuracy?

taehyunzzz: <p>This is more of a accuracy/algorithm question. I would appreciate some empirical/theoretical advices.</p>
<p>Hi, I’m trying to pretrain a ViT model on CIFAR100 (100 classes) from a random pretrained checkpoint I got from huggingface model hub. The model was good for Imagenet validation (80+% accuracy).</p>
<p>Because the pretrained weights were trained on Imagenet (1k classes), I had to re-initialize the final classifier layer of the pretrained model and fine-tune for the 100 classes for CIFAR100. Initially I thought only fine-tuning the re-initialized classifier layer and freezing the rest can save a lot of training time. So I went on with this strategy and it worked. I was able to get 80% test accuracy in 20 epochs and the accuracy did not really improve afterwards.</p>
<p>Afterwards, I “un-froze” the ViT base model parameters and finetuned further. My initial thinking was that the model should begin with 80% accuracy and improve with more epochs of finetuning. However, the accuracy tanked at the first epoch. The accuracy trend for this finetuning showed that it was similar to training a ViT from scratch.</p>
<p>My question here is, shouldn’t finetuning an unfrozen model be as good as when the model was partially-frozen? Does unfreezing the partially frozen layers for further finetuning harm accuracy and not help with faster convergence? Should partially unfreezing the model layers be considered as creating a whole new model?</p>

ptrblck: <p>I guess your learning rate was too large to fine tune the entire model and might have catapulted it away from the pretrained parameters. This is a common issue when fine-tuning a model and you might need to experiment with different learning rates and check at which point the accuracy decreases significantly.</p>

taehyunzzz: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> , I should try out different learning rates as you suggested. Meanwhile, I did found some references saying that different layers have different learning sensitivities so unfreezing a model for finetuning could actually be slower than finetuning from scratch. If I can’t find a good learning rate, I think I will just begin from scratch.</p>
How to access the parameter of one class in another?

ADONAI_TZEVAOT: <p>I have 2 classes A and B. How can class B access parameters that are in class A?</p>

ptrblck: <p>It cannot directly access class members unless it has access to the object. Otherwise you might need to define static variables.</p>

ADONAI_TZEVAOT: <p>How to give access to its object ? Please could you type a sample code?</p>

ptrblck: <p>Sure, here is a small example:</p>
<pre><code class="lang-python">class A(object):
    def __init__(self, B):
        self.B = B
        print(self.B.internal_attribute)
        
class B(object):
    def __init__(self):
        self.internal_attribute = 2.8
        
b = B()
a = A(b)
# 2.8
</code></pre>
CXX ver for LibTorch versions

MoscowskyAnton: <p>Hi!<br>
I am working on ROS Noetic C++ package, which uses libtorch, but I can’t even build it.<br>
I think problem in mismatches between libtorch CXX versions and other ROS libraries, which are compiles only when 11 is on. Is there some documentation where I can check which CXX is used by libtorch versions?<br>
Many thanks!</p>

ptrblck: <p>The current <code>main</code> branch needs a compiler supporting C++17 as seen <a href="https://github.com/pytorch/pytorch#prerequisites">here</a>.</p>
How to rotate batch of images by a batch of angles?

Fleyderer: <p>I have a Tensor of images of size [B, C, H, W] and I want to rotate each of them by a specific angle from the Tensor of angles of size [B].</p>
<p>For example, I have Tensor of images of size [2, 1, 3, 3] and angles of size [2]:</p>
<pre><code class="lang-auto">t = torch.Tensor([
    [[[0, 0, 0],
      [2, 0, 1],
      [0, 0, 0]]],
    [[[0, 2, 0],
      [0, 0, 0],
      [0, 1, 0]]]])

r = torch.Tensor([-180, -90])
</code></pre>
<p>result should be:</p>
<pre><code class="lang-auto">torch.Tensor([
    [[[0, 0, 0],
      [1, 0, 2],
      [0, 0, 0]]],
    [[[0, 0, 0],
      [1, 0, 2],
      [0, 0, 0]]]])

</code></pre>
<p>How can I do it?</p>

Fleyderer: <p>I have already solved this problem but i want to share my solution with community</p>
<pre><code class="lang-auto">def rotate(x: torch.Tensor, degree: torch.Tensor) -&gt; torch.Tensor:
    """
    Rotate batch of images [B, C, W, H] by a specific angles [B] 
    (counter clockwise)

    Parameters
    ----------
    x : torch.Tensor
      batch of images
    angle : torch.Tensor
      batch of angles

    Returns
    -------
    torch.Tensor
        batch of rotated images
    """
    angle = degree / 180 * torch.pi
    s = torch.sin(angle)
    c = torch.cos(angle)
    rot_mat = torch.stack((torch.stack([c, -s], dim=1),
                           torch.stack([s, c], dim=1)), dim=1)
    zeros = torch.zeros(rot_mat.size(0), 2).unsqueeze(2)
    aff_mat = torch.cat((rot_mat, zeros), 2)
    grid = F.affine_grid(aff_mat, x.size(), align_corners=True)
    x = F.grid_sample(x, grid, align_corners=True)
    return x
</code></pre>
DistributedDataParallel didn't sync param gradients across ranks in a process group?

Ziheng_Zhang: <p>The background is that I’m trying to parallelize a serial of module forwards in each worker in distributed data parallel training. Since autograd does not support crossing process boundaries, I came across an idea that I could spawn sub-workers in each worker, initialize new process groups, and use queues to feed input data and gradients from outside workers to sub-workers for forward and backward within. Since the set of modules I’m trying to do parallel forward are the same, I assume that the same module parameters across all sub-workers with the same rank (the rank in sub-worker) should have the same gradients after backward. However, in my experiment below, I observed that the gradients vary across sub-workers with the same rank in the same process group. Did I do something wrong?</p>
<pre><code class="lang-auto">import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel
import torch.nn as nn
from torch.nn.init import uniform_
from loguru import logger
import os
import torch
import time

world_size = 2
sub_world_size = 4


def init_fn(module):
    if isinstance(module, nn.Linear):
        uniform_(module.weight)
        

def main_worker(rank):
    logger.info(f"enter main worker: {rank}")
    dist.init_process_group("nccl", rank=rank, world_size=world_size)
    torch.cuda.set_device(rank)
    out_queues = [Queue() for _ in range(sub_world_size)]
    grad_queues = [Queue() for _ in range(sub_world_size)]
    model = nn.Linear(1, 1, bias=False).apply(init_fn)
    in_tensors = [torch.tensor([(rank + 1) * 10 + sub_rank], dtype=torch.float32) for sub_rank in range(sub_world_size)]
    logger.info(f"main worker before {rank=} model {model.weight.grad=}")
    ctx = mp.spawn(sub_worker, nprocs=sub_world_size, args=(rank, model, in_tensors, out_queues), join=False)
    ctx.join()
    time.sleep(5)
    logger.info(f"main worker after {rank=} model {model.weight.grad=}")



def sub_worker(rank, parent_rank, model, in_tensors, out_queues):
    dist.init_process_group("nccl", rank=parent_rank, world_size=world_size, init_method=f'tcp://127.0.0.1:{29502+rank}')
    in_tensor, out_queue = in_tensors[rank], out_queues[rank]
    out_tensor = DistributedDataParallel(model.cuda(parent_rank))(in_tensor.cuda(parent_rank))
    out_tensor_detached = out_tensor.detach()
    # out_queue.put(out_tensor_detached)
    out_tensor.backward()
    logger.info(f"sub worker: {rank=}, {parent_rank=}, {model.weight.grad=}")


if __name__ == "__main__":
    os.environ["MASTER_ADDR"] = "127.0.0.1"
    os.environ["MASTER_PORT"] = "29501"
    ctx = mp.spawn(main_worker, nprocs=world_size, args=(), join=False)
    ctx.join()
</code></pre>
How to compute mean of a tensor in place given the array of dimensions in Pytorch C++?

nkg114mc: <p>I try to call the tensor’s in place mean function given an array of dimensions in libtorch (Pytorch C++ lib):</p>
<pre><code class="lang-auto">inline at::Tensor mean(const at::Tensor &amp; self, at::OptionalIntArrayRef dim, bool keepdim=false, c10::optional&lt;at::ScalarType&gt; dtype=c10::nullopt)
</code></pre>
<p>Just curious that how to construct the <code>at::OptionalIntArrayRef</code> type argument given a vector&lt;int64_t&gt;?</p>
<p>For example, I have a 3-dimension tensor x. I want to call <code>x.mean({1, 2})</code>, but instead of using initialized_list, I hope the {1, 2} can come from a vector&lt;int64_t&gt; input. How to write such a statement?</p>
Peak memory utilization in CPU?

Abhranta: <p>Hi, I am trying to calculate the peak memory utilization in pyych. While using CUDA, I can do torch.cuda.max_memory_allocated() to get the peak memory utilization. But there is no such option for CPU. For this reason, I am using “psutil.Process().memory_info().rss” to get the memory utilization. But this ends up giving me results that are wildly different from cuda memory usage. What can I do here ?</p>
Double argument type conflict for c++ extension for torchscript

ethany2580: <p>Hi,<br>
I notice an error when i try to add a c++ extension to torchscript, i think the reason is that i have two arguments in double (since torchscript supports double not float), but after i compile them, it gives me this following error:</p>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  Inferred operator schema for a C++ kernel function doesn’t match the expected function schema.<br>
operator: my_namespace::linear_custom<br>
expected schema: my_namespace::linear_custom(Tensor input, Tensor weight, Tensor bias, double alpha, double beta) → (Tensor)<br>
registered at kernels/bindings.cpp:25<br>
inferred schema: (Tensor _0, Tensor _1, Tensor _2, float _3, float _4) → (Tensor _0)<br>
registered at kernels/bindings.cpp:25<br>
reason: Type mismatch in argument 4: double vs float</p>
<p>I have make sure my C++ part does not have float at all…not sure why this happens</p>
<p>BTW previously i run this kernel successfully with pybind11 build in eager mode, im just trying to upgrading it so that i can use graph mode, so I replace the pybind part to:</p>
<p>TORCH_LIBRARY(my_namespace, m) {<br>
m.def(“linear_custom(Tensor input, Tensor weight, Tensor bias, double alpha, double beta) → Tensor”, &amp;linear_custom);<br>
}</p>
How to convert numpy.ndarray with data type np.complex128 to a torch tensor?

Gautam_Phadke: <p>I have a variable named <strong>feature_data</strong> is of type numpy.ndarray, with every element in it being a complex number of form x + yi. How do I convert this to Torch tensor?</p>
<p>When I use the following syntax: <strong>torch.from_numpy(feature_data)</strong>, it gives me an error saying<br>
<strong>can’t convert np.ndarray of type numpy.complex128. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.</strong></p>
<p>I want my torch tensor to have complex numbers in it after conversion from numpy.ndarray.</p>

Henry_Chibueze: <p>Try this:  torch.Tensor(np.array([ ]), dtype=torch.cfloat)</p>

biendltb: <p>In PyTorch 2.0, the version with “torch.tensor” worked for me but not “torch.Tensor”</p>
<pre><code class="lang-python">torch.tensor(np.array([]), dtype=torch.cfloat)
</code></pre>
Getting Loss of Individual Points

Maxerature: <p>I need to obtain the loss of individual points, but I would prefer to do so in batches of size greater than 1.  I have attempted to do so using</p>
<p><code>loss = torch.nn.functional.nll_loss(output, label, reduction='none')</code></p>
<p>but I suspect this to be incorrect.  Although I do get a feasible loss for each point, the process in which these losses are used is performing significantly worse than I’d expect, and I believe this to be due to this step being performed incorrectly.</p>

ptrblck: <p>Could you describe why using <code>reduction='none'</code> is wrong? It will return the loss for each sample of the batch without taking the default <code>'mean'</code> reduction.</p>

Maxerature: <p>It may well be correct, I’m just trying to locate the point of failure in my system, and I thought I may have been using the incorrect method, but if <code>reduction='none'</code> truly results in per-element losses, then the failure must be elsewhere.</p>

ptrblck: <p>You can verify it quickly by comparing the raw loss (with a <code>mean</code> reduction) against the already averaged loss as seen here:</p>
<pre><code class="lang-python">batch_size = 64
nb_classes = 10

output = torch.randn(batch_size, nb_classes, requires_grad=True)
output = output.log_softmax(dim=1)
label = torch.randint(0, nb_classes, (batch_size,))

loss_raw = torch.nn.functional.nll_loss(output, label, reduction='none')
loss = torch.nn.functional.nll_loss(output, label)

print(loss_raw.mean())
# tensor(2.6052, grad_fn=&lt;MeanBackward0&gt;)

print(loss)
# #tensor(2.6052, grad_fn=&lt;NllLossBackward0&gt;)
</code></pre>
<p>and it should return the same value.</p>
RuntimeError: Too many open files... Custom Dataloader

MeiMei_Liu: <p>Hi,</p>
<p>I’m unsure why my code is crashing with the error message (RuntimeError: Too many open files. Communication with the workers is no longer possible.). It works when I only return the variables id, damage, growth_stage, and image_path (string variables), but it fails when I add the variable extent (an integer). How can I resolve this, and why is it happening with integers? Could this be a bug?</p>
<p>Pytorch 2.1.0.dev20230828+cu121<br>
Pandas 2.0.3<br>
PIL 9.3.0</p>
<p>Update: This error is not present in Pytorch 1.13.0</p>
<p>Thanks in advance</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader
from PIL import Image

import os
import random
import pandas as pd


class DroughtDamageDataset(Dataset):
    def __init__(self, df_path, shapes=(384,)):
        self.df_path = df_path
        if len(shapes) == 1:
            self.shapes = (shapes[0], shapes[0])
        elif len(shapes) == 2:
            self.shapes = shapes

        self.df = pd.read_csv(self.df_path)

        if 'train' in self.df_path.lower():
            self.mode = 'train'
            self.root_path = '../dataset/train'
        else:
            self.mode = 'test'
            self.root_path = '../dataset/test'

        self.save_dir = f"../dataset/images/{shapes[0]}"

        if not os.path.exists(self.save_dir):
            os.makedirs(self.save_dir)

    def __len__(self):
        return len(self.df)

    def __getitem__(self, idx):
        row = self.df.iloc[idx]
        image_path = os.path.join(self.root_path, row['filename'])
        # image = Image.open(image_path) \
        #     .convert('RGB') \
        #     .resize(self.shapes)

        save_dir_path = f"{self.save_dir}/{row['ID']}.JPG"
        # image.save(save_dir_path)

        if self.mode == 'train':
            return {
                'id': row['ID'],
                'damage': row['damage'],
                'growth_stage': row['growth_stage'],
                'extent': row['extent'],
                f'image_{self.shapes[0]}': save_dir_path[3:]
            }

trainset = DroughtDamageDataset('../dataset/Train.csv', shapes=(384, 384))

dataloader = DataLoader(
    trainset,
    shuffle=False,
    num_workers=4,
    batch_size=1
)


</code></pre>