akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Gradient through image processor

emei: <p>So in my code i am computing the gradient of the image given as input to a resnet. I use AutoImageProcessor from transformers library. Is it that the gradient will not flow through AutoImageProcessor ? For example in below;</p>
<p>from transformers import AutoImageProcessor</p>
<p>image_processor = AutoImageProcessor.from_pretrained(“microsoft/resnet-18”)</p>
<p>model_input = image_processor(image, return_tensors=“pt”)</p>
<p>outputs = resnet(**model_input, output_hidden_states=True)</p>
<p>When i backpropogate the loss from the model, i obtain gradient at model_input (using model_input.grad), however image.grad gives me an error " The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won’t be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. "</p>

ptrblck: <p><code>model_input</code> is an intermediate activation and thus not a leaf-tensor as the warning describes. Did you try to call <code>.retain_grad()</code> on it as suggested?</p>

emei: <p>Yes , i tried retain_grad() but still get the same issue.</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue, please?</p>
Why is drop_last False by default

st862: <p>By default, drop_last is set to false, so the final batch is retained. I understand that situations with complicated models or small datasets require all the available data to be utilised, and an incomplete final batch can still contribute valuable information.</p>
<p>However, many datasets contain anomalous samples and potentially incorrect labels. The gradients for these anomalous points are likely to differ greatly from the dataset average but usually would make a minimal difference when included in a large batch. An update on a final mini-batch that is smaller than the others and contains too many anomalous points can drastically differ from the global optimum and often increases the loss for the next update causing noisy spikes in the loss over time.</p>
<p>Standard practice is to train over many epochs, shuffling the mini-batches each time. The model will see all the points many times, even if the final mini-batch is discarded in each epoch. Therefore, I believe that the risks of keeping the final minibatch far outweigh any benefit and I can not understand why drop_last = false by default in PyTorch with no warning.</p>

ptrblck: <aside class="quote no-group" data-username="st862" data-post="1" data-topic="188338">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/st862/48/63928_2.png" class="avatar"> st862:</div>
<blockquote>
<p>Therefore, I believe that the risks of keeping the final minibatch far outweigh any benefit and I can not understand why drop_last = false by default in PyTorch with no warning.</p>
</blockquote>
</aside>
<p>I disagree and claim it would be misleading to drop samples from the <code>DataLoader</code> by default.<br>
The user specifies a <code>Dataset</code> and expects to load all samples. The same applies to other arguments such as <code>shuffle</code>. While the standard approach would be to shuffle the dataset setting it to <code>True</code> by default would again confuse a lot of users.</p>
Cuda tensor interact with cpu matrix with gradient retained

Garvey: <p>I have a vector <code>a</code> produced by neural model which need to interact with a huge matrix <code>M</code>. Since <code>M</code> is large, I have to do the computation in cpu device. In this case, I wonder if the gradient can be retained and backwarded on cuda device.</p>
<p>Below is an example. I am looking for solution such that <code>a_cuda.grad</code> has the same gradients as <code>a_cpu.grad</code>.</p>
<pre><code class="lang-auto">a_cuda = torch.randn([1, 512], requires_grad=True).to("cuda")
a_cpu = torch.randn([1, 512], requires_grad=True).to("cpu")

M = torch.randn([512, 100000], requires_grad=False) # loaded in cpu device, dont need update

out_cuda = (a_cuda.cpu() @ M).sum()
out_cuda.backward()

out_cpu = (a_cpu @ M).sum()
out_cpu.backward()

print(a_cuda.grad) # None
print(a_cpu.grad)
</code></pre>

ptrblck: <p>You are ignoring a valid warning:</p>
<pre><code class="lang-python">UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
</code></pre>
<p>which is why <code>a_cuda.grad</code> is <code>None</code>.<br>
Fix it by creating <code>a_cuda</code> on the GPU directly: <code>a_cuda = torch.randn([1, 512], requires_grad=True, device="cuda")</code> and it’ll work.</p>

Garvey: <p>Thanks for the response. Since <code>a_cuda</code> is an output tensor from a model (i.e. <code>a_cuda=model(input)</code>), I fail to find a way to creat it on GPU directly. Is there is any other workaround? Thanks.</p>

ptrblck: <p>If <code>a_cuda</code> is a non-leaf tensor call <code>.retain_grad()</code> on it as suggested in the warning.</p>
What's the difference between symbolic_trace and Tracer

MartinZhang: <p>What’s the difference between <code>torch.fx.symbolic_trace</code> and <code>torch.fx.Tracer().trace</code> ?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torchvision.models.resnet import resnet18

m = resnet18()

m1 = torch.fx.symbolic_trace(m)

m2 = torch.fx.Tracer().trace(m)
</code></pre>
<blockquote>
<p>type(m1)  &lt;class ‘torch.fx.graph_module.GraphModule.<strong>new</strong>..GraphModuleImpl’&gt;</p>
</blockquote>
<blockquote>
<p>type(m2)  &lt;class ‘torch.fx.graph.Graph’&gt;</p>
</blockquote>

grpollak: <ul>
<li><code>torch.fx.symbolic_trace</code>  is a wrapper around a call to<code> fx.Tracer.trace</code> to constructs a <code>torch.fx.GraphModule</code> that holds a <code>torch.fx.Graph</code> attribute.</li>
<li><code>torch.fx.Tracer().trace()</code> calls the tracer only to construct only the FX graph:</li>
</ul>
<pre><code class="lang-auto">graph : torch.fx.Graph = torch.fx.Tracer().trace(m)
</code></pre>
<p>See also <a href="https://pytorch.org/docs/stable/fx.html#writing-transformations" class="inline-onebox" rel="noopener nofollow ugc">torch.fx — PyTorch 2.0 documentation</a></p>
Need Internship Job In Machine learning (Remote)

mathematics: <p>Hi,</p>
<p>I’m graduate student, Currently working as full stack java software developer   . I’ve had deep connection to Python and pytorch in Past , later stopped due to 0 jobs in my country or i didnt even hired as ML intern.  I want to switch career to Again machine learning , pytorch related . What you can get from me is I’m very  hard worker  and very honest .</p>
<p>Thank you for your help .</p>
Training loss remained almost unchanged during training

Hi_Hieu: <p>Hi.<br>
I am training a model.<br>
This is my code to train the model.</p>
<pre><code class="lang-auto">    optimizer = optim.SGD(model.parameters(), lr=args.learning_rate)
    criterion = InfoNCELoss(temperature=0.1)
    scheduler = optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.1)

    log_iter = 100
    
    for e in range(1, args.epoch + 1):
        model.train()
        t0 = time.time()
        total_train_loss = 0

        for i, batch in enumerate(train_dataloader):
 
            optimizer.zero_grad()
            sentence1_embedding, sentence2_embedding = model(batch["sentence1"], batch["sentence2"])
            train_batch_loss = criterion(sentence1_embedding, sentence2_embedding, batch["sentence1_label"], batch["sentence2_label"])

            total_train_loss += train_batch_loss.item()
            train_batch_loss.backward()
            optimizer.step()
            scheduler.step()
</code></pre>
<p>This is my training loss:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c5f2196c012d04528ebb99110553d4a7f5000b7.png" alt="Screenshot from 2023-09-14 17-33-54" data-base62-sha1="1LrxVSzzcwvNFdh77Bfq2Ik5xFJ" width="465" height="135"></p>
<p>Training loss remained almost unchanged during training.<br>
Please help me to solve this.</p>
<p>Thank!</p>
From torchvision import datasets throws error

Nagaraj_Trivedi: <p>Hi, I tried importing datasets from torchvision using the below instruction and it gave the error<br>
python3.6<br>
Python 3.6.9 (default, Mar 10 2023, 16:46:00)<br>
[GCC 8.4.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>from torchvision import datasets<br>
/home/mistral/.local/lib/python3.6/site-packages/torchvision/io/image.py:11: UserWarning: Failed to load image Python extension:<br>
warn(f"Failed to load image Python extension: {e}")</p>
</blockquote>
</blockquote>
</blockquote>
<p>May I know what could be the reason. An early reply will definitely help me.</p>

ptrblck: <p>You are not seeing an error but only a warning pointing out the image extension wasn’t built and you thus might not be able to use <code>torchvision.io</code> methods.</p>
The designing of Neural network

yerongguang.2000: <p>For example, to design a Net, the input feature is an M-dimensional vector, the output is a D-dimensional vector, and then calculate the loss based on the value of the D-dimensional vector. When the output vector value is limited to a discrete value (such as [0, 1, 2, 3, 4]), Net does not seem to be updated, because the input feature forward propagation outputs the values of discrete D-dimensional vectors almost identically. Net is written like this:</p>
<p>class Net(torch.nn.Module):<br>
def <strong>init</strong>(self, n_dim, n_obj, problem):<br>
super(Net, self).<strong>init</strong>()<br>
self.n_dim = n_dim<br>
self.n_obj = n_obj<br>
self.problem = problem<br>
self.fc1 = nn.Linear(self.n_obj, 256)<br>
self.fc2 = nn.Linear(256, 256)<br>
# self.fc3 = nn.Linear(256, self.n_dim*5)</p>
<pre><code>    self.fc3 = nn.Linear(256, self.n_dim)
def forward(self, pref):

    x = torch.relu(self.fc1(pref))
    x = torch.relu(self.fc2(x))
    x = self.fc3(x)

    x = torch.sigmoid(x) * 5
    x = torch.floor(x)


    return x.to(torch.float64)
</code></pre>

yerongguang.2000: <p>Any problems in my code? I want to know why that neural network is don’t work.</p>

ptrblck: <p>The <code>torch.floor</code> operation will return zero gradients and defined <a href="https://github.com/pytorch/pytorch/blob/7f7f6267e99f8dd5990e43dd2301ac37bb66e299/tools/autograd/derivatives.yaml#L690-L692">here</a> and seen here:</p>
<pre><code class="lang-python">x = torch.randn(10, 10, requires_grad=True)

y = torch.floor(x)
y.mean().backward()

print(x.grad)
# tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
#         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
</code></pre>
<p>Working with a discrete output thus won’t work and you should use common approaches of e.g. using logits for multi-class classification use cases etc.</p>
Unet requires automatic_optimization=False but then validation_step is not invoked (Lightning)

pgyurov: <p>I am using Pytorch Lightning to train a UNet model for a binary segmentation task.</p>
<p><strong>Model</strong></p>
<pre><code class="lang-auto">import lightning.pytorch as pl
import segmentation_models_pytorch as smp
import torch.nn as nn

class UNet(pl.LightningModule):
    def __init__(self, num_classes):
        super().__init__()

        self.model = smp.Unet(encoder_name='efficientnet-b2', 
                              encoder_weights='imagenet', 
                              classes=num_classes, 
                              activation='sigmoid')
    
        self.loss_fn = nn.BCEWithLogitsLoss()
        self.automatic_optimization = True
        self.save_hyperparameters()

    def configure_optimizers(self) -&gt; Any:
        return super().configure_optimizers()

    def training_step(self, batch, batch_idx):
        img, gt = batch
        output = self.model(img)
        loss = self.loss_fn(output, gt)
        self.log('train_loss', loss, on_epoch=True, on_step=True)
        return loss

    def validation_step(self, batch, batch_idx):
        img, gt = batch
        output = self.model(img)
        loss = self.loss_fn(output, gt)
        self.log('val_loss', loss, on_epoch=True, on_step=True)

</code></pre>
<p>When I have <code>automatic_optimization</code> set to <code>True</code> I get this error:</p>
<pre><code class="lang-auto">File "/Users/me/.pyenv/versions/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 200, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
</code></pre>
<p>When I have <code>automatic_optimization</code> set to <code>False</code> the Lightning invokes the <code>training_step</code> successfully, but the <code>validation_step</code> never gets called.</p>
<p><strong>What’s the suggested approach when using Lightning?</strong> I’ve been reading the <a href="https://lightning.ai/docs/pytorch/stable/model/manual_optimization.html#manual-optimization" rel="noopener nofollow ugc">Manual Optimization</a> section but not sure if it’s right to define the validation step inside <code>training_step</code>.</p>

pgyurov: <p>I found the issue. Setting <code>automatic_optimization</code> to <code>False</code> is the way to go (correct me if I am wrong).</p>
<p>The reason my <code>validation_step</code> was not being called was because my validation dataset was returning 0 items <img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=12" title=":man_facepalming:" class="emoji" alt=":man_facepalming:" loading="lazy" width="20" height="20"></p>
How to set BCELoss with weight in pytorch lightning

Aravind_Sundaresan: <p>Hi:</p>
<p>I’m training a classifier with pytorch lightning. I have created a LightningDataModule and LightningModule. Where the dataset was balanced, I set the criterion (to compute loss) in the <strong>init</strong>() function like this</p>
<pre><code class="lang-auto">class LightningClassifier(pl.LightningModule):
    def __init__(self, model=None, **kwargs): 
        super().__init__()
        self.criterion = torch.nn.BCELoss()
</code></pre>
<p>However, I now have a datamodule that has an unbalanced training dataset and I want to use <code>BCELoss(weight=weight)</code> like this.</p>
<pre><code class="lang-auto">class LightningClassifier(pl.LightningModule):
    def __init__(self, model=None, **kwargs): 
        super().__init__()
        weight = datamodule.weight("train")
        self.criterion = torch.nn.BCELoss(weight=weight)
</code></pre>
<p>In the above example, it is not possible to do this since when I’m initializing the LightningModule, I do not have access to the datamodule. Where would I set the <code>criterion</code> value in the LightningModule?</p>
<p>I’d appreciate pointers to this.</p>
<p>Thanks,<br>
Aravind.</p>

gphilip: <p>I can’t figure this out with just the code that you shared.</p>

Aravind_Sundaresan: <p>Hi:</p>
<p>Thanks for the response. My question is a conceptual one so I didn’t want to add too much code initially, but I edited my question and I hope it is clear now.</p>
<p>Thanks,<br>
Aravind.</p>

Terbe_Daniel: <p>Hello,<br>
Really good question! I am pondering the same problem. Do you manage to solve it?<br>
<a class="mention" href="/u/aravind_sundaresan">@Aravind_Sundaresan</a></p>

pgyurov: <p>You would have to pass the weights as a function to the model during initialisation.</p>
<pre><code class="lang-auto">class LightningClassifier(pl.LightningModule):
    def __init__(self, model=None, weight_func=None, **kwargs): 
        super().__init__()
        weight = weight_func()
        self.criterion = torch.nn.BCELoss(weight=weight)
</code></pre>
<p>Suppose you’re using a <code>config.yaml</code> file, you could do this like this:</p>
<p><em>config.yaml</em></p>
<pre><code class="lang-auto">model:
  class_path: my.module.LightningClassifier
  init_args:
    model: .....
    weight_func: my.module.my_weight_func
</code></pre>
<p>And in <code>my/module.py</code> you can do:</p>
<pre><code class="lang-auto">def my_weight_func(x,y,z):
    ...
    return weights
</code></pre>
How to set "-CMAKE_CXX_FLAGS=release" when writing CPP-extension

scirocc19900305: <p>am I right?:</p>
<pre><code class="lang-auto">from setuptools import setup, Extension
from torch.utils import cpp_extension
import os
name = 'vecenv'
root_folder = os.path.dirname(os.path.abspath(__file__))
print('root_folder', root_folder)
setup(name=name,
      ext_modules=[cpp_extension.CppExtension(name, sources=['ENV.cpp'],  # source can only contain cpp file
                                              include_dirs=[root_folder,
                                                            root_folder+'/include/third_party/concurrentqueue',
                                                            root_folder+'/include',
                                                            ],
                                              extra_compile_args=['-std=c++17','-O3','-march=native','-CMAKE_CXX_FLAGS=release' ])
                   ],
      cmdclass={'build_ext': cpp_extension.BuildExtension})

</code></pre>
Update model weights with another model and add them both to a common computation graph

avyavkumar: <p>Hi,</p>
<p>Let’s say I have model A and model B. I store model A’s parameters using <code>ParameterDict</code>, compute model B’s parameters using model A parameters and load those new weights into model B’s <code>state_dict</code>. Is there a way to ensure model A’s parameters (parameters in the <code>ParameterDict</code>) are present in the computation graph such that when I do a backward pass, model B and model A are both updated?</p>
<p>Essentially, it would be great if I can do something like</p>
<pre><code class="lang-auto">for name, param_A, param_B in named_parameters(): 
    param_B = param_B - param_A*constant
</code></pre>
<p>such that param_B’s value is updated in place and param_A is added to the computational graph of param_B.</p>
<p>(For anyone familiar with MetaSGD, I want to implement a version of it from scratch).</p>
<p>Thanks,<br>
Avyav</p>

avyavkumar: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> if you get the time, would love to hear your thoughts on this - thank you so much!</p>
Tensor pretty repr in Jupyter

alugowski: <p>Folks using Jupyter might find this handy.</p>
<p>I’m the author of a utility called <code>matrepr</code> ( <a href="https://pypi.org/project/matrepr/" class="inline-onebox" rel="noopener nofollow ugc">matrepr · PyPI</a> ) that can pretty-format sparse and dense tensors to string, HTML, and Latex, with Jupyter integration.</p>
<p>PyTorch’s native string repr works well for dense tensors, but less so for sparse tensors where one must manually decode the internal arrays. MatRepr will pretty-print to a nice 2D matrix, or a coordinates+value tuple if more than 2 dimensions. Hybrid tensors are also supported. CSR/CSC are explicitly decoded so the output is instantaneous no matter how large the tensors are.</p>
<p>See a demo comparing native repr with MatRepr on several types of tensors at <a href="https://nbviewer.org/github/alugowski/matrepr/blob/main/doc/demo-pytorch.ipynb" class="inline-onebox" rel="noopener nofollow ugc">Jupyter Notebook Viewer</a></p>
<p>Feedback welcome!</p>
<p>Example:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d15880fc9b8b03bc57dd7565b8eb2ff0c7af576.png" data-download-href="https://discuss.pytorch.org/uploads/default/6d15880fc9b8b03bc57dd7565b8eb2ff0c7af576" title="Screenshot 2023-09-13 at 22-42-04 Notebook on nbviewer"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d15880fc9b8b03bc57dd7565b8eb2ff0c7af576_2_690x348.png" alt="Screenshot 2023-09-13 at 22-42-04 Notebook on nbviewer" data-base62-sha1="fz0aRNqOG6y2kPL5F9Owk9MTQXQ" width="690" height="348" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d15880fc9b8b03bc57dd7565b8eb2ff0c7af576_2_690x348.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d15880fc9b8b03bc57dd7565b8eb2ff0c7af576_2_1035x522.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d15880fc9b8b03bc57dd7565b8eb2ff0c7af576_2_1380x696.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-09-13 at 22-42-04 Notebook on nbviewer</span><span class="informations">1597×806 41.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Getting "trying to backward through graph second time" error while implementing stateful LSTM

saj: <p>I am implementing Stateful LSTM:<br>
class StatefulFraudLSTM(nn.Module):<br>
def <strong>init</strong>(self, num_features, hidden_size=100, hidden_size_lstm=100, num_layers_lstm=3, dropout_lstm=0, batch_size=128):<br>
super(StatefulFraudLSTM, self).<strong>init</strong>()<br>
# Parameters<br>
self.num_features = num_features<br>
self.hidden_size = hidden_size<br>
self.hidden_size_lstm = hidden_size_lstm<br>
self.num_layers_lstm = num_layers_lstm<br>
self.batch_size = batch_size</p>
<pre><code>    # Representation learning part
    self.lstm = nn.LSTM(num_features, hidden_size_lstm, num_layers_lstm, batch_first=True, dropout=dropout_lstm)

    # Representation to hidden
    self.fc1 = nn.Linear(hidden_size_lstm, hidden_size)
    self.relu = nn.ReLU()

    # Hidden to output
    self.fc2 = nn.Linear(hidden_size, 1)
    self.sigmoid = nn.Sigmoid()

    # Initialize hidden and cell states
    self.hidden = self.init_hidden()

def init_hidden(self):
    # Initialize hidden and cell states with zeros
    h0 = torch.zeros(self.num_layers_lstm, self.batch_size, self.hidden_size_lstm).to(device)
    c0 = torch.zeros(self.num_layers_lstm, self.batch_size, self.hidden_size_lstm).to(device)
    return (h0, c0)

def forward(self, x):

    representation, self.hidden = self.lstm(x.transpose(1, 2), self.hidden)

    hidden_state = self.hidden[0][-1]
    out = self.fc1(hidden_state)
    out = self.relu(out)

    out = self.fc2(out)
    out = self.sigmoid(out)

    return out
</code></pre>
<p>Getting below error:</p>
<pre><code class="lang-auto">RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
</code></pre>

ptrblck: <p>You might need to <code>.detach()</code> the hidden states before passing them to the model.</p>

saj: <p>I am learning and implementing PyTorch. How can I do that. Could you please help me.</p>
LSTM Quantization

mahuichao: <p>I am learning quantization of LSTM in Pytorch. from the  source code I can not see the implementation of “static” quantization of LSTM, the last function I can see is as follows:torch.quantized_lstm,  I can not see the implementation of this function.By the way, are there any paper about dynamic quantization of LSTM ?<br>
Thanks.</p>

jerryzh168: <p>I’m not aware of papers about dynamic quantization for LSTM. here is an example for statically quantize LSTM: <a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/fx/test_quantize_fx.py#L4782" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/test/quantization/fx/test_quantize_fx.py#L4782</a> in fx graph mode quantization,  dynamically quantize LSTM examples can be found here: <a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/eager/test_quantize_eager_ptq.py#L1333" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/main/test/quantization/eager/test_quantize_eager_ptq.py#L1333</a> (with eager mode quantization)</p>
Can the output of operator QuantizedConv2d is fp32?

TingfengTang: <p>Hi ,<a class="mention" href="/u/jerryzh168">@jerryzh168</a> , The operator QuantizedConv2d’s output  is now int8, is there a configuration to force it to float32(e.g. delete the obersver at the output, intuitively).Because if there is just one conv2d layer appears, its quantized version operator QuantizedConv2d will requant the output to int8, then a dequantize layer followed to convert the output to float32, it may decrease the performance.  what i want in formulation expression is:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/0/20752d860d8b35adda002fcc2658453a1a302dbb.png" alt="quant" data-base62-sha1="4D8kxRAxOQPDcvFuDEG6DdEKzoL" width="637" height="93"></p>
<p>Thanks</p>

HDCharles: <p>the op itself only has one type of output. the QuantizedConv2d’s output is always quint8 so your options are to not quantize it or to dequantize the quantized output</p>

jerryzh168: <p>this is not possible in old flows, but possible in the new flow: <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2.0 Export Post Training Static Quantization — PyTorch Tutorials 2.0.1+cu117 documentation</a>, but we need to onboard more backends before this can be used by modeling users. which backend are you targeting right now?</p>

TingfengTang: <p>Thanks for your reply, but i can’t import XNNPACKQuantizer in pytorch distributed version 2.01.  The backend i used is qnnpack,with QATconfig(weight_per_tensor_symmetric,input_per_tensor_symmetric,output_float32)</p>

jerryzh168: <p>we have 2.1 release coming up soon, could you wait a bit and upgrade to 2.1? you’ll get XNNPACKQuantizer there.</p>
Trying to perturb parameters of BNN

armal: <p>I am currently trying to perturb the parameters of my trained bayesian neural net by a random amount using this function:</p>
<p>copied_bnn = copy.deepcopy(bnn)</p>
<p>perturbation_scale = 100</p>
<p>for name, param in copied_bnn.named_parameters():<br>
if ‘mu’ in name:<br>
rho_name = name.replace(‘mu’, ‘rho’)<br>
sigma_param = torch.exp(bnn.state_dict()[rho_name] / 2.0)<br>
perturbation = torch.normal(0.0, sigma_param * perturbation_scale)<br>
param.data += perturbation</p>
<p>The problem is, no matter how much I perturb the parameters by, the evaluation on the test set stays the same.</p>

ptrblck: <p>Add <code>float32_max</code> as a quick test and check if your model outputs invalid values. If not, you might not be properly manipulating the parameters.</p>

armal: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="188278">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>float32_max</code></p>
</blockquote>
</aside>
<p>Just tried this and the NN is still outputting the same test accuracy. Is it something to do with the computational graph being copied across as well? This is the full code for the copying across:</p>
<p>import copy</p>
<p>copied_bnn = copy.deepcopy(bnn)</p>
<p>copied_bnn = copied_bnn.to(torch.device(device))</p>
<p>perturbation_scale = 3.4e38</p>
<p>for name, param in copied_bnn.named_parameters():<br>
if ‘mu’ in name:<br>
rho_name = name.replace(‘mu’, ‘rho’)<br>
sigma_param = torch.exp(bnn.state_dict()[rho_name] / 2.0)<br>
perturbation = torch.normal(0.0, sigma_param * perturbation_scale)<br>
with torch.no_grad():  # Detach from computation graph<br>
param.data += perturbation</p>
<p>correct = 0<br>
total = 0</p>
<p>with torch.no_grad():<br>
for inputs, labels in test_loader:<br>
inputs, labels = inputs.to(device), labels.to(device)<br>
outputs = copied_bnn(inputs.view(1, -1))<br>
_, predicted = torch.max(outputs.data, 1)<br>
total += labels.size(0)<br>
correct += (predicted == labels).sum().item()</p>
Looking for correct version of pytorch for RTX 4090, CUDA 12.2

Chethan_Chandran: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a73561da25052e3e59950180afa1a4c7b0b47e8.png" alt="image" data-base62-sha1="aCCrgVJSh8Nqo5YWbUQqGAtoeM0" width="257" height="112"><br>
This is my pytorch version and my GPU is “NVIDIA GeForce RTX 4090”</p>
<p>When I try to train my model, Kernel dies with the below output<br>
C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:93: block: [62,0,0], thread: [115,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
Need help !!</p>

ptrblck: <p>The error message points towards an indexing error and is unrelated to the used PyTorch binary.<br>
Narrow down the failing indexing operation and fix it.</p>

Chethan_Chandran: <p>Yes, indeed it was.<br>
Fixed it. Thanks.</p>

zabi: <aside class="quote no-group" data-username="Chethan_Chandran" data-post="1" data-topic="185788">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chethan_chandran/48/62935_2.png" class="avatar"> Chethan_Chandran:</div>
<blockquote>
<p>NVIDIA GeForce RTX 4090</p>
</blockquote>
</aside>
<p>How to build pytorch-cuda=12.2 ? my GPU is NVIDIA GeForce RTX 4090 and I have nvidia toolkit 12.2 installed on my windows 11.</p>

ptrblck: <p>You would need to build from source using your locally installed CUDA toolkit. Alternatively, you can download the nightly binaries using CUDA 12.1 Update1.</p>

zabi: <p>Thank you. I couldn’t find the nightly build for 12.2.</p>
Help loading a Yolov5 model with toch.hub.load()

csheerer: <p>I am trying to run inference on a model created via Yolov5. I am trying to load the model with weights using torch.hub.load() and am getting an error</p>
<h3>Command</h3>
<pre><code class="lang-python">model = torch.hub.load('ultralytics/yolov5', 'custom', path='/yolov5/runs/train/exp9/weights/best.pt', force_reload=True)
</code></pre>
<h3>Error</h3>
<pre><code class="lang-sh">File ~/.cache/torch/hub/ultralytics_yolov5_master/hubconf.py:33, in _create(name, pretrained, channels, classes, autoshape, verbose, device)
     17 """Creates or loads a YOLOv5 model
     18 
     19 Arguments:
   (...)
     29     YOLOv5 model
     30 """
     31 from pathlib import Path
---&gt; 33 from models.common import AutoShape, DetectMultiBackend
     34 from models.experimental import attempt_load
     35 from models.yolo import ClassificationModel, DetectionModel, SegmentationModel

File ~/.cache/torch/hub/ultralytics_yolov5_master/models/common.py:38
     35     os.system('pip install -U ultralytics')
     36     import ultralytics
---&gt; 38 from ultralytics.utils.plotting import Annotator, colors, save_one_box
     40 from utils import TryExcept
     41 from utils.dataloaders import exif_transpose, letterbox

File /opt/conda/lib/python3.10/site-packages/ultralytics/utils/__init__.py:25
     22 import yaml
     23 from tqdm import tqdm as tqdm_original
---&gt; 25 from ultralytics import __version__
     27 # PyTorch Multi-GPU DDP Constants
     28 RANK = int(os.getenv('RANK', -1))

ImportError: cannot import name '__version__' from 'ultralytics' (unknown location)
</code></pre>
<p>I am running in AWS Sagemaker with the Pytorch 2.0.0 Python 3.10 image and here are my versions<br>
torch: 2.0.0<br>
ultralytics 8.0.177</p>
<p>Any advice would be helpful.</p>
Clarity on nondeterminism with torch.nn.functional.pad

daturkel: <p>Hello, I’d like to use <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.pad.html#torch.nn.functional.pad" rel="noopener nofollow ugc">torch.nn.functional.pad</a> in the design of a neural network that will be trained on a GPU. I noticed the note on the docs page for <code>pad</code>:</p>
<blockquote>
<p>When using the CUDA backend, this operation may induce nondeterministic behaviour in its backward pass that is not easily switched off. Please see the notes on <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">Reproducibility</a> for background.</p>
</blockquote>
<p>However, when I checked the Reproducibility page, I wasn’t sure which—if any—of the sections applied to <code>pad</code>. There are several particular notes about cuDNN for convolutional operations, and I suspect that many CNN designs leverage <code>pad</code>, but my model does not use convolutions. How can I determine if the nondeterminism warning is relevant to me?</p>
CUDA 11.8 package version issues and CUDA 11.7 version issue

Brit_W: <p>My custom built model trains fine when using CUDA 11.8 with these packages and versions:<br>
<strong>pytorch-triton==2.1.0+e6216047b8</strong><br>
<strong>torch==2.1.0.dev20230804+cu118</strong><br>
<strong>torchaudio==2.1.0.dev20230804+cu118</strong><br>
<strong>torchvision==0.16.0.dev20230804+cu118</strong></p>
<p>Today I decided to create a new conda environment and followed the instructions to install pytorch 2.0 11.8. I notice that there are different versions now:<br>
<strong>pytorch-triton==2.1.0+6e4932cda8</strong><br>
<strong>torch==2.2.0.dev20230913+cu118</strong><br>
<strong>torchaudio==2.2.0.dev20230913+cu118</strong><br>
<strong>torchvision==0.17.0.dev20230913+cu118</strong></p>
<p>My model no longer trains and I get this error when trying to train it (this is an error I did not receive before).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edb39334a51fafe1f11beb13bc3f53320ef155af.png" data-download-href="https://discuss.pytorch.org/uploads/default/edb39334a51fafe1f11beb13bc3f53320ef155af" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/d/edb39334a51fafe1f11beb13bc3f53320ef155af.png" alt="image" data-base62-sha1="xUNTGHP68TrNl4yC5utneIAoc23" width="690" height="438" data-dominant-color="2C2C2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">784×498 19 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>“message”: “backend=‘inductor’ raised:\nCalledProcessError: Command ‘[’/usr/bin/gcc’, ‘/tmp/tmpypr9vj84/main.c’, ‘-O3’, ‘-I/SD5/people/s1208875/miniforge3/envs/torch_test/lib/python3.9/site-packages/triton/common/…/third_party/cuda/include’, ‘-I/SD5/people/s1208875/miniforge3/envs/torch_test/include/python3.9’, ‘-I/tmp/tmpypr9vj84’, ‘-shared’, ‘-fPIC’, ‘-lcuda’, ‘-o’, ‘/tmp/tmpypr9vj84/cuda_utils.cpython-39-x86_64-linux-gnu.so’, ‘-L/lib64’, ‘-L/lib64’]’ returned non-zero exit status 1.\n\nSet TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information\n\n\nYou can suppress this exception and fall back to eager by setting:\n    import torch._dynamo\n    torch._dynamo.config.suppress_errors = True\n”</strong></p>
<p>This becomes a problem if other people need to reproduce my conda environment that has CUDA 11.8. I read in another post that CUDA 11.7 is more stable so I will try to download and use that one instead.</p>
<p>Any thoughts as to what happened with CUDA 11.8?</p>

ptrblck: <p>The error points to <code>triton</code> in the paths, so I would start by looking into it instead of CUDA.</p>
Embedding_bag operator on GPU

Rishabh_Jain: <p>Hello,</p>
<p>Nvidia suggests to use <a href="https://github.com/NVIDIA/TensorRT" rel="noopener nofollow ugc">TensorRT</a> framework for a performant inference deployment. For DLRM (DL based Recommendation Systems) inference on GPU, I have the following questions:</p>
<ul>
<li>
<p>Does TensorRT modify the backend (CUDA/C++ source code) of Embedding bag operator or it uses the exact same vanilla PyTorch CUDA kernels?</p>
</li>
<li>
<p>What are the benefits of using vanilla PyTorch over TensorRT for DLRM inference?</p>
</li>
</ul>
<p>Please let me know your comments. Thanks</p>
PyTorch C++ neural network output given as a product of two numbers

Ramansh_Sharma: <p>Hi. I just started using PyTorch C++ fully. I set up a simple neural network that can evaluate a given tensor. When evaluating the neural network, the output (which I set to be a scalar, for only one input in this case) is shown as follows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbe46b4dd38c40a1a0741131e06cc615c9a44527.png" data-download-href="https://discuss.pytorch.org/uploads/default/bbe46b4dd38c40a1a0741131e06cc615c9a44527" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbe46b4dd38c40a1a0741131e06cc615c9a44527.png" alt="image" data-base62-sha1="qOaFI5nlZgJ5hFN5b78AGHxF1lB" width="690" height="35" data-dominant-color="2B2B2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1502×78 3.89 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To test whether this output was a scalar Tensor (or a symbolic thing), I instead outputted this output plus a c++ float, and the result was then a correct scalar Tensor being shown with one number. Does anyone know why C++ by default shows the output in this format?</p>
<p>Thank you.</p>

Ramansh_Sharma: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, apologies for tagging you promptly here. Do you happen to know of this behavior or know someone who can help me understand this. Thank you.</p>

ptrblck: <p>I’m not sure which formats you are comparing, but they are defined <a href="https://github.com/pytorch/pytorch/blob/55f956f1d2e92e3f04d9a7e99c485450321323c7/aten/src/ATen/core/Formatting.cpp#L69">here</a> in case you want to check it.</p>

Ramansh_Sharma: <p>Thank you. I see at the end of the function that it returns <code>return std::make_tuple(scale, sz);</code> a scale and a size in a tuple. This shouldn’t affect any arithmetic operations one would do with tensors, right?</p>

ptrblck: <p>Yes, this won’t change the internal value but just formats it for easier readability during printing.</p>
How to start loading a dataset from a specific index

Bahman_Rouhani: <p>Suppose I have a very large dataset, so large that it would take a considerable time to go through them even once. I’m training a model on this dataset and at some point before even completing one epoch on the whole dataset, my training might be stopped or killed. I want to be able to resume the training process. The problem is, I don’t want to start going through the dataset from start, i.e. I want to resume the training from the exact last sample that the model was trained on.</p>
<p>At the moment, I can’t figure out how to set the index for Dataloader object.</p>
<p>Is there a way to do that?</p>
The trainloader size remains unchanged when concatenating data beforehand

Scbas_scias: <p>I have a np.array of type uint8 of face images as input. Then I create two different transforms and apply them to the input. However, I encounter some problems with the dataloader:<br>
The output size is not what I expected. It is 3136 regardless of whether I use augmentation or not.</p>
<pre><code class="lang-auto">train_data = torch.from_numpy(train_data)
train_data = rearrange(train_data, 'b h w c -&gt; b c h w')

transform1 = T.Compose([
            T.TrivialAugmentWide(),
            lambda x : x.float(),
            T.Normalize(mean=[0.485, 0.456, 0.406], 
                        std=[0.229, 0.224, 0.225])
    ])
transform2 = T.Compose([
            T.ToTensor(),
            T.Normalize(mean=[0.485, 0.456, 0.406], 
                        std=[0.229, 0.224, 0.225])
    ])

train_dataset = ConcatDataset(datasets=[
            CustomDataset(data = train_data, labels = train_lab, transforms=transform1),
            CustomDataset(data = train_data, labels = train_lab, transforms=transform2)

    ])
train_dataloader = DataLoader(train_dataset, batch_size=parameters['batch_size'], sampler=sampler)
</code></pre>

ptrblck: <p>The dataset will usually apply the transformation on-the-fly on each sample in its <code>__getitem__</code> method. You are using a <code>CustomDataset</code>, which is not defined in the posted code snippet, so you would need to double check it.<br>
If you are also sticking to the common approach then the size won’t change if a transformation is used or not.</p>
Num_workers in data loader

zaa: <pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, x_path, y_path):
        self.x_path = x_path
        self.y_path = y_path
        self.x_file_list = sorted(glob.glob(os.path.join(x_path, '*.npy')))
        self.y_file_list = sorted(glob.glob(os.path.join(y_path, '*.npy')))
        
        if len(self.x_file_list) != len(self.y_file_list):
            raise ValueError("Number of X and Y files must match.")
        
    def __len__(self):
        return len(self.x_file_list)

    def __getitem__(self, idx):
        x_data = np.load(self.x_file_list[idx])
        y_data = np.load(self.y_file_list[idx])
        x_tensor = torch.tensor(x_data)
        y_tensor = torch.tensor(y_data)
        return x_tensor, y_tensor



train_dataset= CustomDataset(path_train_data_x,path_train_data_y)
val_dataset=CustomDataset(path_val_data_x,path_val_data_y)


train_loader=DataLoader(dataset=train_dataset,batch_size=200,num_workers=4)
val_loader=DataLoader(dataset=val_dataset,batch_size=200,num_workers=4)
</code></pre>
<p>so this is my dataset class and dataloader, i am facing bottleneck problem in batch loading when i used default value for num_workers.</p>
<p>But as i increase num_workers , and try to get a single batch using next(iter(train_loader)) its just running like an infinite loop, and not loading a single batch.</p>
<p>I am using cpu with 16 cores, i7 gen 11. And i am loading images which are saved in .npy() of size 128x128.</p>

ptrblck: <p>Add the <code>if-clause</code> guard as described <a href="https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection">here</a> and check if this would fix the issue.</p>
Layer Normalization affects in Batch Overfit Testing

alex_sole: <p>I am using pytorch geometric to predict specific chemical properties (regression task) using the GraphGPS repo (<a href="https://github.com/rampasek/GraphGPS" class="inline-onebox" rel="noopener nofollow ugc">GitHub - rampasek/GraphGPS: Recipe for a General, Powerful, Scalable Graph Transformer</a>).</p>
<p>For sanity check, we did an overfitting test with only one molecule (same sample for training and val), and we realized that when Layer Normalization is active at a certain point of the training, the validation stops, but training continues decreasing. If we deactivate the Layer Norm, this effect is not happening. Why does the loss differ from val to train if it is the same sample?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/4/f4db749fb4ce2af4022058bf4fd37bf1c81cb462.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f4db749fb4ce2af4022058bf4fd37bf1c81cb462" title="image (1)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4db749fb4ce2af4022058bf4fd37bf1c81cb462_2_690x323.jpeg" alt="image (1)" data-base62-sha1="yW6GpV5xVRSHzmsYww6oaEum4Ui" width="690" height="323" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4db749fb4ce2af4022058bf4fd37bf1c81cb462_2_690x323.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4db749fb4ce2af4022058bf4fd37bf1c81cb462_2_1035x484.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/4/f4db749fb4ce2af4022058bf4fd37bf1c81cb462_2_1380x646.jpeg 2x" data-dominant-color="FDFCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image (1)</span><span class="informations">1920×899 69 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As far as I know, Layer Norm is the same during training and validation. It is not like batch norm that uses running mean and var from the training samples.</p>
<p>Also, we want to understand why the loss is not going to zero the loss with only one sample, no matter the learning rate or the number of epochs. Is it common in regression tasks? Do you assume some error? (Yes, we know it’s a low error, but it is an overfitting test with only one sample). Is it because of the model’s capacity or a bug in the code? Why the loss cannot go lower?</p>