akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom QAT using ao.nn.qat modules, is this a valid approach?

username: Khumbaba
cooked: <p>Hello,</p>
<p>Since the model I’m using has a GRU, which is not QAT ready in torch yet, and since I need to do PQT later in TfLite, I’ve decided to try to do QAT using ao.nn.qat modules while modifying them to add activation FakeQuant without going through traditional (buggy?) route.</p>
<p>An extract of how I’ve implemented the module:</p>
<pre><code class="lang-auto">class GRU_qat(nn.GRU):
    def __init__(
        self,
        input_size: int,
        hidden_size: int,
        num_layers: int = 1,
        bias: bool = True,
        batch_first: bool = False,
        dropout: float = 0.0,
        bidirectional: bool = False,
        qconfig=None,
        device=None,
        dtype=None,
    ) -&gt; None:
        factory_kwargs = {"device": device, "dtype": dtype}
        super().__init__(input_size, hidden_size, num_layers, bias, batch_first, dropout, bidirectional, **factory_kwargs)
        assert qconfig, "qconfig must be provided for QAT module"
        self.qconfig = qconfig
        self.activation_fake_quant = qconfig.activation(factory_kwargs=factory_kwargs)
        self.weight_fake_quant_1 = qconfig.weight(factory_kwargs=factory_kwargs)
        self.weight_fake_quant_2 = qconfig.weight(factory_kwargs=factory_kwargs)
    
    def forward(self, input, hx=None):
...
            result = _VF.gru(self.activation_fake_quant(input), batch_sizes, hx, [self.weight_fake_quant_1(self._flat_weights[0]), self.weight_fake_quant_2(self._flat_weights[1]),
                                          self._flat_weights[2], self._flat_weights[3]], self.bias,
                             self.num_layers, self.dropout, self.training, self.bidirectional)
...

</code></pre>
<p>The process I followed was simple: Load the pretrained weights to the original model with its QAT equivalent modules then train normally at a low learning rate for a few epochs on the training dataset. I’ve tested both weight only (without activation fake quant) and full QAT. I didn’t see much of an improvement but I already have strong quantized performance.</p>
<p>Now my question is: is my method valid? Backpropagation didn’t give me an error so I assumed torch handled the fake quantization well, but did it really? I’m aware that I ignored the case of layer fusion, batch norm and math ops. Are their impact great? I assumed the weight fake quant will be the most important.</p>
<p>Thank you!</p>

username: jerryzh168
cooked: <p>I’d recommend using <a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch main documentation</a> since we plan to deprecate older eager and fx flows, we’ll make a post in the coming months about this</p>
Absence of qint32 in torch.ao.quantization.utils.weight_is_quantized

username: tjdgns0928
cooked: <p>I am using FX quantization with a custom backend and custom layers.</p>
<p>Quantization with <code>qint8</code> is working well. However, when I tried to quantize the model using <code>qint32</code>, the layer was not quantized during the <code>convert_fx</code> step.</p>
<p>Upon investigation, I found that the issue is due to the absence of <code>torch.qint32</code> in the <code>torch.ao.quantization.utils.weight_is_quantized</code> function.</p>
<pre><code class="lang-auto">def weight_is_quantized(qconfig):
    """ Given a qconfig, decide if the weight needs to be
    quantized or not
    """
    return weight_dtype(qconfig) in [
        torch.quint8,
        torch.qint8,
        torch.float16,
        torch.quint4x2,
        torch.uint8,
        torch.int8,
        torch.int16,
        # torch.qint32  &lt;&lt;&lt; this is missing.
    ]
</code></pre>
<p>Is there a specific reason why qint32 is not included as a target for weight quantization?</p>

username: jerryzh168
cooked: <p>feel free to add, although we are moving to a new flow now which has easier support for dtypes since we are using native pytorch dtypes like uint8/int8/int16/int32 instead special qint8 dtypes</p>
<p><a href="https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/main/quantization.html#prototype-pytorch-2-export-quantization</a></p>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation when using ReLU(inplace=False)

username: Mole_Turner
cooked: <p>I’m running <code>models.py</code> from <a href="https://github.com/raymondxyy/strfnet-IS2020/tree/master" rel="noopener nofollow ugc">this</a> repository. Running the code as is gives me the error <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [24, 32, 16, 65]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>What’s confusing is that the ReLU instance is using the default <code>inplace=False</code>. If I change to using the LeakyReLU activation with <code>negative_slope = 0.</code> (giving me plain ol’ ReLU) the code runs without an error.</p>
<p>Any ideas what may be causing this? I’m happy to have found a solution, but I’d quite to know what might be causing it to break in the first place.</p>

username: ptrblck
cooked: <p>The issue is most likely not related to the <code>relu</code> operation itself, but to another op which is manipulating the <code>relu</code> output:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

relu = nn.ReLU()

x = torch.randn(10, 10, requires_grad=True)

# works
out = relu(x)
out = out * 2
out.mean().backward()
x.grad = None

# fails
out = relu(x)
out *= 2 # inplace 
out.mean().backward()
# RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [10, 10]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead.
</code></pre>
<p>since <code>relu</code> needs its output to compute the <code>dgrad</code>.</p>

username: Mole_Turner
cooked: <p>In this instance, changing from using nn.ReLU(x) to nn,LeakyreLU(x) should still result in the same error being thrown?</p>

username: ptrblck
cooked: <p>No, since <code>leaky_relu</code> needs its input for the gradient calculation as seen <a href="https://github.com/pytorch/pytorch/blob/b75bb64eb4bf6fea36b8ee7c7a67074bdaa7c69a/tools/autograd/derivatives.yaml#L2147-L2149">here</a> while <code>relu</code> needs its output as seen <a href="https://github.com/pytorch/pytorch/blob/b75bb64eb4bf6fea36b8ee7c7a67074bdaa7c69a/tools/autograd/derivatives.yaml#L2091-L2093">here</a>.</p>

username: Mole_Turner
cooked: <p>That’s interesting. Might be outside the scope of this thread, but what is the reason for that difference?</p>
<p>With respect to the output though, would you recommend hunting down and fixing the stray inplace operation, or will using <code>leaky_relu</code> with <code>negative slope = 0</code>. achieve the same outcome?</p>
Torchvision,Torchaudio and Xavier Nx

username: Halil_Huseyin_Calisk
cooked: <p>Hello, I want to process images with a GPU using the NVIDIA Xavier NX AI kit. I have installed torch. But where can I download torchvision and torchaudio in a way that is compatible with the GPU? Is there a direct download method using pip, instead of Docker?</p>

username: ptrblck
cooked: <p>You could build these packages as described <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a> or you could use a container which ships with all pre-built libs.</p>
RuntimeError: Function 'ExpBackward' returned nan values in its 0th output.

username: yuansongzeng
cooked: <p>MeanAct = lambda x: torch.clamp(torch.exp(x), 1e-5, 1e6)</p>
<p>Traceback (most recent call last):<br>
loss.backward()<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
RuntimeError: Function ‘ExpBackward’ returned nan values in its 0th output.</p>
<p>I used the function MeanAct as the last activation function and the backward, An error has occurred. as seen in above</p>

username: albanD
cooked: <p>Hi,</p>
<p>Can you share a small code sample we can use to reproduce the error please? This looks surprising indeed.</p>

username: ptrblck
cooked: <p>Probably unrelated, but just for the sake of completeness:<br>
we had apparently an issue in <code>1.4</code> for <code>pow</code> in <a href="https://discuss.pytorch.org/t/function-powbackward1-returned-nan-values-in-its-1th-output/87138/7">this thread</a>.</p>
<p><a class="mention" href="/u/yuansongzeng">@yuansongzeng</a> which PyTorch version are you using?</p>

username: tonyr
cooked: <p>I have the same error, so I found this post.  For anyone else that find this, the clamping should be done to prevent under/overflow before exp() is called.  Something like <code>torch.exp(x.clamp(-11,14))</code></p>
Pycharm Terminal VS Terminal

username: Diego_Lovison
cooked: <p>Hey, I wanted to share something I’ve encountered:</p>
<p><strong>Issue 1:</strong><br>
When running my test from the PyCharm terminal with the DataLoader configured as follows, everything works fine:</p>
<pre data-code-wrap="bash"><code class="lang-bash">--num-workers 10 --batch-size 20
</code></pre>
<p>However, if I run the same test from my regular terminal, I get this error:</p>
<pre><code class="lang-auto">Bus error. It is possible that dataloader's workers are out of shared memory. Please try to raise your shared memory limit.
</code></pre>
<p>Why the training is working with Pycharm terminal and not in my regular terminal?</p>
<p><strong>Issue 2:</strong><br>
In a similar scenario, the test works fine in the PyCharm terminal but fails in my regular terminal with this error:</p>
<pre><code class="lang-auto">Too many open files
</code></pre>
<p><strong>What I’ve discovered so far:</strong><br>
It seems PyCharm’s terminal applies some pre-configurations. For example:</p>
<p>In the PyCharm terminal:</p>
<pre data-code-wrap="bash"><code class="lang-bash">$ ulimit -n
1048576
</code></pre>
<p>In my regular terminal:</p>
<pre data-code-wrap="bash"><code class="lang-bash">$ ulimit -n
1024
</code></pre>
<p>Is there a recommended list of configurations to set up before starting training? It seems like there should be! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Trying to move everything to MPS on M2 mac

username: cekcreator
cooked: <p>Hey yall! I’m a college student who is trying to learn how to use pytorch for my lab and I am going through the pytorch tutorial with the MNIST dataset. I was trying to move “operations” over to my GPU with both. But I think I am missing moving more that just the model over. I have checked some posts on here and stack overflow but I cant find anything that I haven’t tried yet. Any help or direction would be appreciated!</p>
<pre><code class="lang-auto">torch.set_default_device("mps")
# and the below 
class NeuralNetwork(nn.Module):
# code for the class here
model = NeuralNetwork()
model.to("mps")
</code></pre>

username: ptrblck
cooked: <p>Moving the model and inputs to your device should be enough and execute the model on it.</p>