akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Custom computation of only some gradients

programehr: <p>Hi,</p>
<p>I’m gonna define a function with two arguments. I need to implement the derivateive w.r.t. one of them myself, but let the autograd engine calculate the other derivateive. I’ve tried grad, vjp, jacfwd, and jacrev from the functorch package to leave the second gradient to autograd. They do work, but are too slow. Here is an example:</p>
<pre><code class="lang-auto">def f(x, y):
    return x**2 * torch.sin(y)


class F(Function):
    @staticmethod
    def forward(ctx, x, y):
        output = f(x, y)  # x**2 * torch.sin(y)
        ctx.save_for_backward(x, y)
        return output

    @staticmethod
    def backward(ctx, grad_output):
        x, y = ctx.saved_tensors
        grad_y = x**2 * torch.cos(y)
        grad_x = jacfwd(f, argnums=0)(x, y)
        return grad_x, grad_y
</code></pre>
<p>The above code is just a simple example. I defined how to differentiate w.r.t. y, but had autograd do this for x.<br>
My actual code is longer and more costly to run, though. Is there any way I could do this?</p>
Error RuntimeError: Boolean value of Tensor with more than one value is ambiguous

Veronika_Oktafia: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>. Could you help me with these issue? Pleasee, help me</p>
<p>fig, axes = plt.subplots(6,6, figsize=(24,24))<br>
for image, label, pred, ax in zip (feature, target, preds, axes.flatten()):<br>
ax.imshow(image.permute(1,2,0).cpu())<br>
font = {“color”: ‘r’} if label != pred else {“color”:‘g’}<br>
label, pred = label2cat [label.item()], label2cat[preds.item()]<br>
ax.set_title(f"Label: {label} | Pred: {pred}", fontdict= font);<br>
ax.axis(“off”);</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9b87ed0e5d5c0a1e06447c142d27f7e377ab412.png" data-download-href="https://discuss.pytorch.org/uploads/default/d9b87ed0e5d5c0a1e06447c142d27f7e377ab412" title="Screenshot (747)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9b87ed0e5d5c0a1e06447c142d27f7e377ab412_2_690x387.png" alt="Screenshot (747)" data-base62-sha1="v42Tdud5HJXuuiA5QLraIBXJKnw" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9b87ed0e5d5c0a1e06447c142d27f7e377ab412_2_690x387.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/9/d9b87ed0e5d5c0a1e06447c142d27f7e377ab412_2_1035x580.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9b87ed0e5d5c0a1e06447c142d27f7e377ab412.png 2x" data-dominant-color="656464"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (747)</span><span class="informations">1366×768 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Please help me, how to fixed the error</p>

ptrblck: <p>Based on the error message it seems either <code>label.item()</code> or <code>preds.item()</code> is causing the error as one of them (or both) is containing multiple elements and thus cannot be converted to a scalar value.<br>
I would also generally recommend not to tag specific users as it could demotivate other users to post a valid response.</p>
Layers support for MaskedTensor type

pablosmig: <p>Hi there!</p>
<p>I’m currently working with tensors of different shapes. I was looking at the MaskedTensor prototypes which looks like a great solution and everything I needed to concatenate tensors of different shapes while masking the elements that are padded in inputs with shapes different than max shape in the corresponding dimension.</p>
<p>However, when I forward them through standard pytorch layers I get errors like:</p>
<pre><code class="lang-auto">In convolution:
TypeError: no implementation found for 'torch._ops.aten.convolution.default' on types that implement __torch_dispatch__: [&lt;class 'torch.masked.maskedtensor.core.MaskedTensor'&gt;]

In linear:
TypeError: no implementation found for 'torch._ops.aten.mm.default' on types that implement __torch_dispatch__: [&lt;class 'torch.masked.maskedtensor.core.MaskedTensor'&gt;]
</code></pre>
<p>I saw there is support for most (if not all. for the linear layer at least) tensor operations required to build these layers. Yet, the layers fail to operate on MaskedTensor. I assume this is because the layers are not implemented using the basic operations supported by MaskedTensor.</p>
<p>If this is the case, is there any work going into making layers support MaskedTensor. This would greatly relief users having to worry about ill-conditioned gradients when a tensor is mostly padded with zeros just using autograd. (NB: My original issue is div-by-zero in a weight normalization backward hook).<br>
Otherwise, is how can we use MaskedTensor in basic layers such as convolution and linear?</p>
<p>Thanks! PyTorch is an amazing tool. 99% of the time I can do what I want without overthinking (everything has been tought of already for my use cases). This is just in the 1% wishlist <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"> and probably being worked on already.</p>

soulitzer: <p>You may be interested in NestedTensor which may have support for the operators you need <a href="https://pytorch.org/docs/stable/nested.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nested — PyTorch 2.0 documentation</a>.</p>

pablosmig: <p>This is very interesting, thank you!</p>

Michael_Chao: <p>Hello, I am experiencing the same issue. Have you found a solution yet?</p>
Using torch 2.0 compile (hidet backend)

Nirmai: <p>TransformException: Error when running pytorch model : hidet_backend raised NotImplementedError: The following modules/functions are not supported by hidet yet:<br>
torch.nn.functional.leaky_relu</p>
<p>self.leaky_relu_V_2(tensor_0)</p>
<p>TransformException: Error when running pytorch model : hidet_backend raised NotImplementedError: The following modules/functions are not supported by hidet yet:<br>
torch.nn.ZeroPad2d</p>
Multi-strean Resenet50 or I3D version

Maths_Electronics_Tu: <p>i trained two models based on I3D from mmaction2 config , one for RGB dataset and the second for optical flow , i need to fuse the best models but i need flexibility to fuse them at any layer or final stage classifier , i need design class that take the pretarined model (pth) as base and creat new model ,that i can make choice in which layer i concatenate outputs to feed than one branch , final classifier .</p>
<p>my first is to fuse them before feeding final head as i3d , the second time i will freeze up layers and fuse outputs to train new weights and bias of the same architecture of i3d (inflated resnet50)</p>

ptrblck: <p>Double post from <a href="https://discuss.pytorch.org/t/multistream-resnet50-i3d/182341">here</a> without sufficient information what exactly the question is.</p>

Maths_Electronics_Tu: <p>I need to design model based on mmaction2 framework , so i need to go with config , to build model based on resnet50  multistream , dataset with multi frames types (rgb frames , optical flow frames )</p>
Inquiry about using pre-trained swin_v2_b on imagenet22k

lmc6130: <p>Hello PyTorch community,</p>
<p>I hope this message finds you well. I have been trying to utilize the pre-trained swin_v2_b model on the Imagenet22k dataset (‘<a href="https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12_192_22k.pth" rel="noopener nofollow ugc">https://github.com/SwinTransformer/storage/releases/download/v2.0.0/swinv2_base_patch4_window12_192_22k.pth</a>’), but I have encountered some difficulties. Despite my efforts, I have been unable to resolve the issues of loading pre-trained model from ‘load_state_dict_from_url’ function and instantiating the <code>swin_v2_b</code> model from <code>torchvision.models</code>. However, I am unsure how to proceed from there to address the above challenges.</p>
<p>Could someone kindly guide me on how to properly load the pre-trained swin_v2_b model with the imagenet22k weights, access its input features, modify the architecture if needed, and ensure its successful integration into my project?</p>
<p>Any help, suggestions, or code examples would be greatly appreciated. Thank you in advance for your time and support.</p>
Transforming pretrained ResNet50 to a hierachical model by dividing it into subnetworks

Tm4: <p>For extracting features from images I use pretrained ResNet50 (removing the classifier head). In order to transform the model to a hierarchical model, I want to use the following strategy:</p>
<pre><code class="lang-auto">import torch
from torchvision.models import resnet50, ResNet50_Weights
import torch.nn as nn

class model(nn.Module):
    def __init__(self):
        super(model, self).__init__()

        self.model = resnet50(weights=ResNet50_Weights.IMAGENET1K_V2)
        self.block1 = nn.Sequential(*list(self.model.children())[:5])
        self.block2 = nn.Sequential(*list(self.model.children())[5:6])
        self.block3 = nn.Sequential(*list(self.model.children())[6:7])
        self.block4 = nn.Sequential(*list(self.model.children())[7:8])


    def forward(self, x):   
        
        x1 = self.block1(x)    
        x2 = self.block2(x1)
        x3 = self.block3(x2)
        x4 = self.block4(x3)

        return(x1, x2, x3, x4)
</code></pre>
<p>this is because I need to have the intermediate outputs. My question is that does this approach cause any problem for assigning the pretrained weights to different layers (i.e., self.block2, self.block3 ,…)? Should I edit the source code for my goal or the mentioned strategy is a working idea?</p>
Backward hook not called on JIT functions

AlphaBetaGamma96: <p>Hi All,</p>
<p>I’ve been working with trying to call a <code>register_full_backward_hook</code> on a jitted function yet it’s never called at all. I did notice there’s an issue open about this (<a href="https://github.com/pytorch/pytorch/issues/51969" class="inline-onebox" rel="noopener nofollow ugc">Support backward hooks in JIT · Issue #51969 · pytorch/pytorch · GitHub</a>).</p>
<p>Are there any work-around solution to this?</p>
<p>Any help would be greatly apprecitated! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Tihaw: <p><a href="https://discuss.pytorch.org/t/torchscript-for-the-backward-autograd-graph/81873">"TorchScript has full support for PyTorch’s tape-based autograd. You can call <code>backward()</code> on your tensors if you are recording gradients and it should work. "</a><br>
maybe this will work. I will try.</p>
Forward() got an unexpected keyword argument 'image'

Tanushree_Halder: <hr>
<p>TypeError                                 Traceback (most recent call last)<br>
~\AppData\Local\Temp\ipykernel_12760\4032920361.py in <br>
----&gt; 1 trainer.train()</p>
<p>~\anaconda3\lib\site-packages\transformers\trainer.py in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)<br>
1643             self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size<br>
1644         )<br>
 → 1645         return inner_training_loop(<br>
1646             args=args,<br>
1647             resume_from_checkpoint=resume_from_checkpoint,</p>
<p>~\anaconda3\lib\site-packages\transformers\trainer.py in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)<br>
1936<br>
1937                 with self.accelerator.accumulate(model):<br>
 → 1938                     tr_loss_step = self.training_step(model, inputs)<br>
1939<br>
1940                 if (</p>
<p>~\anaconda3\lib\site-packages\transformers\trainer.py in training_step(self, model, inputs)<br>
2757<br>
2758         with self.compute_loss_context_manager():<br>
 → 2759             loss = self.compute_loss(model, inputs)<br>
2760<br>
2761         if self.args.n_gpu &gt; 1:</p>
<p>~\anaconda3\lib\site-packages\transformers\trainer.py in compute_loss(self, model, inputs, return_outputs)<br>
2782         else:<br>
2783             labels = None<br>
 → 2784         outputs = model(**inputs)<br>
2785         # Save past state if it exists<br>
2786         # TODO: this needs to be fixed and made cleaner later.</p>
<p>~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
1192         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1193                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1194             return forward_call(*input, **kwargs)<br>
1195         # Do not call functions when jit is used<br>
1196         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>TypeError: forward() got an unexpected keyword argument ‘image’</p>
<p>How can I solve this ?</p>

ptrblck: <p>Based on the error message your <code>inputs</code> object seems to contain an <code>image</code> argument which is unexpected in the <code>forward</code> method of your model.<br>
It also seems you are using a higher-level API from HuggingFace, so it’s a bit unclear what inputs are expected. You might thus need to check the model definition and its expected input arguments.</p>
Handling huge dataset ,practical issuses

Btorch: <p>I am currently attempting to train using the laion400m Dataset, but I am encountering numerous issues. The data has been downloaded and is organized in the following manner:</p>
<ul>
<li>00000.parquet</li>
<li>00000.tar</li>
<li>00000_stats.json</li>
</ul>
<p>There are approximately 40,000 triplets of files structured in this manner.</p>
<p>My initial approach involved untaring all the files, which resulted in the following triplets:</p>
<ul>
<li>004137496.txt</li>
<li>004137496.jpg</li>
<li>004137496.json</li>
</ul>
<p>However, I have encountered several problems during this process:</p>
<ol>
<li>Untaring the files is a time-consuming operation.</li>
<li>Untaring the files consumes a significant amount of memory.</li>
<li>Handling a directory containing 400 million files (400m * 3) is extremely challenging.</li>
</ol>
<p>I am seeking advice on the common practices employed when working with datasets of this nature. Has anyone worked with the laion 400m Dataset before? Are there any suggestions or recommendations?</p>
<p>By the way, I attempted to stream the data using Hugging Face, but this method proved to be 30 times slower.</p>
<p>Thank you for any assistance you can provide!</p>
No version satisfies the requirement torch==1.1x.x+cu111

sleepope: <p>I have 2 RTX 3090 GPUs and <code>nvidia-smi</code> outputs the CUDA version is 11.2. I tried to follow the instruction in “Start Locally | PyTorch” but none of this versions returns <code>True</code> for <code>torch.cuda.is_available()</code>.</p>
<p>When I want to further locate my problem, I get <em>No CUDA GPUs are available</em></p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.zeros(1).cuda()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/&lt;username&gt;/miniconda3/envs/&lt;envname&gt;/lib/python3.10/site-packages/torch/cuda/__init__.py", line 217, in _lazy_init
    torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available
</code></pre>
<p>I find a similar problem on StackOverflow: <a href="https://stackoverflow.com/a/62361395/16642369" rel="noopener nofollow ugc">Pytorch says that CUDA is not available</a>, it says I probably should install a PyTorch with a CUDA version below 11.2. I find the following command in <a href="https://pytorch.org/get-started/previous-versions/#linux-and-windows-3" rel="noopener nofollow ugc">Previous PyTorch Versions</a></p>
<pre><code class="lang-bash"># CUDA 11.1
pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>But it gives <em>ERROR: Could not find a version that satisfies the requirement</em>:</p>
<pre><code class="lang-auto">Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.10.1+cu111 (from versions: 1.11.0, 1.11.0+cpu, 1.11.0+cu102, 1.11.0+cu113, 1.11.0+cu115, 1.11.0+rocm4.3.1, 1.11.0+rocm4.5.2, 1.12.0, 1.12.0+cpu, 1.12.0+cu102, 1.12.0+cu113, 1.12.0+cu116, 1.12.0+rocm5.0, 1.12.0+rocm5.1.1)
ERROR: No matching distribution found for torch==1.10.1+cu111
</code></pre>
<p>How to resolve this issue? Any suggestions would be much appreciated.</p>

ptrblck: <p>Install the binaries with the CUDA11.3 or 11.6 runtime and check if you are still hitting the issue.<br>
If so, reinstall your NVIDIA drivers, reboot the node, and rerun your test.</p>

sleepope: <p>Later I found out this is caused by I accidentally run</p>
<pre><code class="lang-bash">export CUDA_VISIBLE_DEVICES=
</code></pre>

jake_gallagher: <p>Can you help me with this?</p>
Custom GRU module for Quantization Aware Training

WillyWonka: <p>I really need to be able to do quantization aware training on GRU layers and PyTorch doesn’t support it yet. However, it seems to support static quantization for LSTM layers through custom modules. So, I used what was done in <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantizable/modules/rnn.py" rel="noopener nofollow ugc">torch/ao/nn/quantizable/modules/rnn.py</a> to make a quantizable version of the GRU layers.</p>
<ol>
<li>
<p>Am I on the right track by following this? From what I read, the only missing part at that point is the <code>from_observed( )</code> method that can be found in <a href="https://github.com/pytorch/pytorch/blob/main/torch/ao/nn/quantized/modules/rnn.py" rel="noopener nofollow ugc">torch/ao/nn/quantized/modules/rnn.py</a></p>
</li>
<li>
<p>What’s the use of calling <code>torch.ao.quantization.prepare( )</code> in the <code>from_float( )</code> method and <code>torch.ao.quantization.convert( )</code> in the <code>from_observed( )</code> method? Isn’t that done when calling <code>prepare_qat( )</code> or <code>convert( )</code> in the main program for example?</p>
</li>
<li>
<p>In the quantization doc, it says “<em>Currently, there is a requirement that ObservedCustomModule will have a single Tensor output, and an observer will be added by the framework (not by the user) on that output. The observer will be stored under the activation_post_process key as an attribute of the custom module instance. Relaxing these restrictions may be done at a future time</em>”. Does it means that the forward method can’t output the output features AND the final hidden state?</p>
</li>
</ol>
<p>Thanks a lot and have a nice day!</p>

jerryzh168: <ol>
<li>Yes, you can modify the quantizable LSTM and implement a similar version for GRU to support static quantization for GRU using custom module API</li>
<li>we are calling prepare/convert in from_float and from_observed because we are relying eager mode quantization to prepare and convert the submodules of the quantizable custom module here</li>
<li>I think the doc is referring to the support in fx graph mode quantization, and quantizable LSTM is actually supported there as well now (<a href="https://github.com/pytorch/pytorch/blob/main/test/quantization/fx/test_quantize_fx.py#L4886" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test/quantization/fx/test_quantize_fx.py at main · pytorch/pytorch · GitHub</a>), if you are using eager mode APIs (e.g. prepare_qat), this shouldn’t be an issue I think.</li>
</ol>
What is the `_replicated_tensor_module` in DDP?

Fei_Liu: <p>I’m looking at the DDP source code <code>train()</code> function (<a href="https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel" rel="noopener nofollow ugc">reference</a>):</p>
<pre><code class="lang-auto">    def train(self, mode=True):
        super().train(mode)
        if self._use_replicated_tensor_module:
            self._replicated_tensor_module.train(mode)  # type: ignore[union-attr]
        return self
</code></pre>
<p>What is the <code>_replicated_tensor_module</code> for?</p>
<p>Context: I’m trying to understand the difference between <code>ddp_model.train()</code> and <code>ddp_model.module.train()</code>.</p>
Registering a hook for a module parameter forward pass instead of a backward pass

HappyNeat: <p>I have a list of select parameters from various parts of a model (type: nn.Module).</p>
<p>I want to get the inputs to the parameters as a forward pass is called on the model.</p>
<p>Is this possible? I only found hooks for the backward pass of a tensor: <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.register_hook.html" class="inline-onebox" rel="noopener nofollow ugc">torch.Tensor.register_hook — PyTorch 2.0 documentation</a></p>

ptrblck: <p>Yes, you could use forward hooks via <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.register_forward_hook"><code>nn.Module.register_forward_hook</code></a>.</p>

HappyNeat: <p>I’m looking for a hook for parameters rather than modules. Something like <code>torch.Tensor.register_forward_hook</code> if it exists.</p>

ptrblck: <p>I don’t think there is one. Why can’t you access the parameters via the module?</p>

HappyNeat: <p>Some modules already have other hooks for other reasons. Is it possible to register multiple hooks for the same module?</p>

ptrblck: <p>Yes, it is possible to use multiple hooks:</p>
<pre><code class="lang-python">lin = nn.Linear(1, 1)
lin.register_forward_hook(lambda m, input, output: print("hook1 for module {}".format(m)))
lin.register_forward_hook(lambda m, input, output: print("hook2 for module {}".format(m)))

x = torch.randn(1, 1)
out = lin(x)
# hook1 for module Linear(in_features=1, out_features=1, bias=True)
# hook2 for module Linear(in_features=1, out_features=1, bias=True)
</code></pre>
Using Pretrained Resnet50 for MASKRCNN_RESNET50_FPN_V2

chokevin8: <p>Hi, as the title suggests, I have a pretrained resnet50 model weights available, and I wanted to use those pretrained weights to fine tune the MASKRCNN_RESNET50_FPN_V2 model from torchvisions. I’ve successfully imported the weights, as it says “all keys matched successfully”. However, in this model, the resnet50 is the backbone weights only, as it also needs the FPN weights. I’m getting an error related to that:<br>
File"…/feature_pyramid_network.py", line 185, in forward<br>
names = list(x.keys())<br>
AttributeError: ‘Tensor’ object has no attribute ‘keys’</p>
<p>I just don’t know how to fix this, do I just follow the implementation <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/mask_rcnn.py#L516" rel="noopener nofollow ugc">here</a>? Thank you!</p>

ptrblck: <p>The input to <code>FeaturePyramidNetwork.forward</code> expects an <code>OrderedDict[Tensor]</code> containing the feature maps for each feature level, which is why the error is raised.<br>
You could refer to <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">this tutorial</a> to see an example usage.</p>

chokevin8: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the quick response and the link to the tutorial, but I still wasn’t able to solve the same error after trying the codes in the tutorial. I basically performed the same code as the tutorial for modifying the model for a new backbone, except that my backbone is loaded with my custom resnet50 weights.</p>
<p>I now understand that the FPN.forward expects the OrderedDict containing the feature maps, but how do I edit the feature maps and how do I decide how many feature maps, and which output size the feature maps should be? Sorry, I’m completely new to instance segmentation/object detection, so I’m not too familiar with this.</p>
Quantization backend selection failed

mahuichao: <p>Hello<br>
I think I came across some problem when selecting quantization backend. From the documents I know pytorch select “x86” by default, but I was learning “fbgemm” code recently, so I change it to “fbgemm”. But the code still use oneDNN as backend, I don’t know why?<br>
My env: Mac M1<br>
Pytorch version: 2.0.0</p>
<p>here is my code:</p>
<pre><code class="lang-auto">    model_fp32_fused.qconfig = torch.ao.quantization.get_default_qconfig('fbgemm')
    model_fp32_prepared = torch.ao.quantization.prepare(model_fp32_fused, inplace=True)
    model_int8 = torch.ao.quantization.convert(model_fp32_prepared)
    model_int8.eval()
    inputs = torch.randn(1, 1, 59, 13) 
    output = model_int8(inputs)
</code></pre>

mahuichao: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/6898d88392b71d6d21cd07b5f5d912ea4f3b46de.png" alt="image" data-base62-sha1="eVj7TVqfkXMYTM8SF67Ivtwh2do" width="584" height="316"><br>
from setup.py I saw this, does this mean that if I want to change the quantization backend, I can’t just change the default_qconfig, I have to recompile the total pytorch source code ? is this right?</p>

jerryzh168: <p>you can do <code>torch.backends.quantized.engine = "fbgemm"</code> to change the backend, here is the doc: <a href="https://pytorch.org/docs/stable/quantization.html#note-for-native-cpu-backends" class="inline-onebox" rel="noopener nofollow ugc">Quantization — PyTorch 2.0 documentation</a></p>

mahuichao: <aside class="quote no-group quote-modified" data-username="jerryzh168" data-post="3" data-topic="182551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerryzh168/48/15217_2.png" class="avatar"> jerryzh168:</div>
<blockquote>
<p>torch.backends.quantized.engine = “fbgemm”</p>
</blockquote>
</aside>
<p>hello thanks for relying, after I change the engine to fbgemm, still error , the error message is:</p>
<blockquote>
<p>RuntimeError: quantized engine FBGEMM is not supported</p>
</blockquote>
<p>what does this mean, My mac book don’t support fbgemm?</p>

jerryzh168: <p>maybe it’s because FBGEMM is not compiled, please make sure USE_FBGEMM is 1 in the generated CMakeLists.txt files</p>
Concatenate two tensor for a linear layer

Amine: <p>So i have one output from a linear layer dimension torch.randn(8, 512, 2048).<br>
i want to concatenate it with another tensor torch.randn(8, 512) to a<br>
linear layer  <code>nn.Linear(2048+512, num_classes)</code>. how to do it?<br>
thx in advance</p>

ptrblck: <p>You won’t be able to directly concatenate these tensors as the second one misses a dimension.<br>
You could <code>unsqueeze</code> the missing <code>dim1</code> and <code>expand</code> the tensor assuming you want to repeat its values <code>512</code> times.</p>

Amine: <p>i have that for every sample input( batchsize = 8)( tensor torch.randn(8, 512)) i have a tensor of 512 which add information and should improve the model prediction. how to integrate this information, i thnk it doesnt make sense to repeat the values</p>

ptrblck: <p>This would work:</p>
<pre><code class="lang-python">a = torch.randn(8, 512, 2048)
b = torch.randn(8, 512)

b = b.unsqueeze(1).expand(-1, 512, -1)

c = torch.cat((a, b), dim=2)
print(c.shape)
# torch.Size([8, 512, 2560])
</code></pre>
Graph saint heterogeneous

cctony1: <p>Hello. I’m wondering if there is a timeline for enabling the conversion of the graphsaint samplers from homo to hetero graphs</p>

ptrblck: <p>I guess this question is for PyG, not PyTorch in general, so I would recommend asking in their GitHub repository so the code owners would see it.<br>
CC <a class="mention" href="/u/rusty1s">@rusty1s</a> for viz</p>

cctony1: <p>Thank you. Good point. Okay, I will.</p>
Kernel crashes when dimension error on an Apple Silicon Device

pypdeveloper: <p>Hello everybody, I have noticed that whenever there is a dimension error with a tensor or a mismatch between the input shape of a model and the shape of the data the kernel crashes. For example, I was running this snippet of code <code>model_2(rand_image_tensor.unsqueeze(0).to(device))</code> and there seemed to be a miss match in the input shapes, there was no error printed out or any kind of warning, the kernel just crashed. I am following the <a href="http://learnpytorch.io" rel="noopener nofollow ugc">learnpytorch.io</a> tutorial and he is running it on google colab, so I wrote the same code that he wrote, changing the device-agnostic code to match with Apple silicon and it worked fine for him but is crashing for me. I am running a conda environment and have installed Jupyter Lab the way that has been shown on the official website, this issue also persists on the VSCode Jupyterlab extension. There is no error displayed just that the kernel crashes. I am running Python 3.11 with the stable Pytorch packages, and I am running this on an M1 Max chip. Is this a bug with all computers? And is there a fix to this, so that when I am building models without a guide I can understand what the error is, and debug it with the proper methods?</p>
<p>Note I am running pytorch version 2.0.1</p>
<p>Thank you so much for you help!!!</p>
In register_full_backward_hook - grad is None for first module

Sergey_Stepanov: <p>Why are gradients not defined in the first module (the last one when backward)?<br>
Thank you.</p>
<pre><code class="lang-auto">import torch, torch.nn as nn

model = nn.Sequential(nn.Linear(3, 2),  nn.Linear(2, 1) )

def hook(module, grad_input, grad_output):        
    for k,p in module.named_parameters():
        print(f"{k:20s} {tuple(p.grad.shape) if p.grad is not None else p.grad }  {tuple(p.data.shape)}")    

model[0].register_full_backward_hook(hook)
model[1].register_full_backward_hook(hook)

x = torch.randn(1, 3)
out = model(x)
out.mean().backward()
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">weight               (1, 2)  (1, 2)
bias                 (1,)  (1,)
weight               None  (2, 3)
bias                 None  (2,)
</code></pre>

Sergey_Stepanov: <p>see: <a href="https://github.com/pytorch/pytorch/issues/56380" rel="noopener nofollow ugc">register_full_backward_hook does not consistently fire #56380</a></p>
Cuda.is_available returns True in consel but False in program

SHANG_JIN: <p>I am a beginner in pytorch and deep learning. When I trying to run the program with pytorch, it shows</p>
<pre><code class="lang-auto">torch._C._cuda_init()
RuntimeError: No CUDA GPUs are available
</code></pre>
<p>Then I added the test code</p>
<pre><code class="lang-auto">print(t.cuda.is_available())
</code></pre>
<p>and it returns False.</p>
<p>But it comes the most wired thing that when I typed line by line in the consel:</p>
<pre><code class="lang-auto">python
&gt;&gt;&gt;import torch
&gt;&gt;&gt; torch.version.cuda
'11.5'
&gt;&gt;&gt; torch.cuda.is_available()
True
</code></pre>
<p>I am also sure that I have GPUs in the server, but why does the program cannot use the GPU?</p>

ptrblck: <p>Often these issues are caused by (accidentally) using different environments and binaries.<br>
Check the <code>torch.__path__</code> in both setups as these could be different.</p>

SHANG_JIN: <p>Thanks very much and I have solved this problem. The key is that the program has stated only the No.2 GPU to be visible but in the server, I only have No.0. I deleted that line and wrote torch.cuda.set_device(0) and it worked.</p>
Incompatible gradient size after using jit.script

jkarolczak: <p>Hi! I have a set of models and some time ago I decided to speed the training up by using <code>torch.jit.script</code> before running the training loop. For most models, it works just fine. But for one model (Implementation of SCINet) I noticed a weird issue:</p>
<p>Jitting works fine, training goes well while using the CPU. However, when I use a CUDA device, for some inputs I get an error like:</p>
<p><code>RuntimeError: Function torch::jit::(anonymous namespace)::DifferentiableGraphBackward returned an invalid gradient at index 1 - got [6, 1, 1] but expected shape compatible with [6, 1, 6] </code></p>
<p>Interestingly, I have many tests (positive path, have been working before) in a pipeline, but only some of them started failing after jitting the model.</p>
<p>The source code is here: <a href="https://pastebin.com/YCH68ApT" class="inline-onebox" rel="noopener nofollow ugc">from typing import Optional, Tuple, Unionimport numpy as npimport torchi - Pastebin.com</a></p>
<p>Is there any way to have more information about errors, to debug the issue?</p>
Torch.where not working for float('nan')

aswamy: <pre><code class="lang-auto">var = torch.tensor([float('nan'), 1.0])

torch.where(var == float('nan'))  gives wrong output?
</code></pre>
<p>(tensor([], dtype=torch.int64),)</p>

ptrblck: <p>This is expected, since <code>NaN == NaN</code> returns <code>False</code> by definition:</p>
<pre><code class="lang-python">x = torch.tensor(float("nan"))
x == x
# tensor(False)

a = np.array(np.nan)
a == a
# False

float("nan") == float("nan")
# False
</code></pre>

aswamy: <p>THanks for the response!<br>
So, there is no direct way to get the indices of NaN in torch. I have to find indirect ways like replacing NaNs with specific values and then doing torch.where() for that replace value?</p>

ptrblck: <p>You could use <code>torch.isnan</code>:</p>
<pre><code class="lang-python">torch.where(torch.isnan(var))
# (tensor([0]),)
</code></pre>

aswamy: <p>Thank you for quick solution!</p>
Image in same set after random_split() in multiple datapipes

avtar: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44a08bb658205d42d175f975e976dd49b6802567.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/44a08bb658205d42d175f975e976dd49b6802567" title="TorchData Diagram_230627_104627"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44a08bb658205d42d175f975e976dd49b6802567_2_689x435.jpeg" alt="TorchData Diagram_230627_104627" data-base62-sha1="9N6r0JLDFD5PksmrR9r3NtxhupV" width="689" height="435" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44a08bb658205d42d175f975e976dd49b6802567_2_689x435.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44a08bb658205d42d175f975e976dd49b6802567_2_1033x652.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44a08bb658205d42d175f975e976dd49b6802567_2_1378x870.jpeg 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">TorchData Diagram_230627_104627</span><span class="informations">2536×1600 243 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With reference to the above diagram, we have a use-case where we are creating separate datapipes for each <code>label_classes</code>.</p>
<p>Let’s say we have an <strong>Image101.jpg</strong></p>
<p>Now this <strong>Image101.jpg</strong> has been annotated with <code>label_cls_1</code> and <code>label_cls_2</code><br>
i.e.,<br>
<strong>Image101.jpg</strong> will be in result-set of DB query for both <code>datapipe_1</code> and <code>datapipe_3</code> as well.</p>
<p>Now, How can we make sure that after <code>random_split()</code> <strong>Image101.jpg</strong> would come either under <strong>TRAIN</strong> set or <strong>TEST</strong> set for both <code>datapipe_1</code> and <code>datapipe_3</code><br>
i.e.,<br>
If <strong>Image101.jpg</strong> comes in <code>test_datapipe_1</code> after <code>random_shuffle()</code> then, it should also come under <code>test_datapipe_3</code> and not <code>train_datapipe_3</code></p>
<p>What would be the recommended way for this?</p>

ptrblck: <p>Depending on the randomness of the splitting approach e.g. by seeding the code could be quite tricky and could easily cause issues. I would recommend to split the dataset once and then to use the split indices to create <code>Subset</code>s for the other datasets, which would make sure no data is leaked from the training into the test set in any pipeline.</p>
Instantiating VGG16 with state dictionary without downloading weights

cltexe: <p>I do have a pth file for <code>vgg16</code> as <code>vgg16-397923af.pt</code>h. I like to instantiate a new vgg16 model with exact same weights in vgg16-397923af.pth file. <code>vgg16 = models.vgg16()</code> always tries to download weights first, however I don’t want to use any downloads just a reference to the pth file. Is this possible?</p>

ptrblck: <aside class="quote no-group" data-username="cltexe" data-post="1" data-topic="182714">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cltexe/48/41817_2.png" class="avatar"> cltexe:</div>
<blockquote>
<p><code>vgg16 = models.vgg16()</code> always tries to download weights first,</p>
</blockquote>
</aside>
<p>That’s not the case as the default argument will randomly initialize the parameters:</p>
<pre data-code-wrap="python"><code class="lang-python">model = models.vgg16()
# no output
model = models.vgg16(pretrained=True)
# Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to ...
</code></pre>
<p>and will store it in <code>~/.cache/torch/hub/checkpoints</code> by default.</p>

cltexe: <p>Thanks ptrblck,<br>
when <code>model = models.vgg16()</code> it download weights, but when <code>model = models.vgg16(pretrained=False)</code> it randomly initializes the parameters.</p>
<blockquote>
<p>pip list | grep torch<br>
torch                 2.0.1<br>
torchaudio            2.0.2<br>
torchvision           0.15.2</p>
</blockquote>

ptrblck: <p>It’s not doing it for me as mentioned in my previous post and also in the <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.vgg16.html">docs</a>:</p>
<blockquote>
<p><strong>weights</strong> (<a href="https://pytorch.org/vision/main/models/generated/torchvision.models.vgg16.html#torchvision.models.VGG16_Weights"><code>VGG16_Weights</code></a>, optional) – The pretrained weights to use. See <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.vgg16.html#torchvision.models.VGG16_Weights"><code>VGG16_Weights</code></a> below for more details, and possible values. By default, no pre-trained weights are used.</p>
</blockquote>
<p>so you would need to come up with a way to reproduce the issue.</p>

cltexe: <p>Well you are correct. Looks like I was absent minded when posting this. It is great to have you here ptrblck, rapidly responding to everything since from the beginning.</p>

ptrblck: <p>Good to hear, it’s indeed not the case as I was struggling to find any way this would be possible. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Is `retain_grad` orthogonal to `retain_graph`?

NightMachinery: <p>Is <code>retain_grad</code> orthogonal to <code>retain_graph</code>? I am a bit confused on their difference. My current understanding is that <code>retain_graph</code> retains <code>.grad_fn</code> attributes, while <code>retain_grad</code> retains <code>.grad</code> attributes.</p>

srishti-git1110: <p><code>retain_graph</code> argument when set to <code>True</code> during a backward call on a tensor <code>x</code> causes autograd NOT to agressively free the saved references to the intermediate tensors in the graph of <code>x</code> that are required for the gradient computation of <code>x</code> wrt some tensor.</p>
<p><code>y.retain_grad()</code> is used to populate the <code>grad</code> attribute of <code>y</code>, which is a non-leaf tensor, when a <code>.backward()</code> call is made – this is the non-default behaviour.</p>

NightMachinery: <blockquote>
<p>the saved references to the intermediate tensors in the graph of <code>x</code> that are required for the gradient computation of <code>x</code> wrt some tensor.</p>
</blockquote>
<p>Where are these references stored? Are they only implicitly stored in the function <code>grad_fn</code>?</p>
Is it safe to use `Tensor.cuda(non_blocking=True)` in a thread?

Fei_Liu: <p>E.g. if I create a thread pool with subsequent operations on async data transfer from host to device (or any async cuda operations in general), will it be problematic? For (a more specific) example: can it happen that a thread local object (say by <code>torch.rand(1_000_000_000).pin_memory()</code>) is passed to async cuda copy then destroyed immediately upon thread deallocation before the async cuda copy call completes?</p>
<p>A related question: if I stick to multi-threading (not multi-processing), is it true that all threads will share the same cuda stream event handles (which I assume is thread-safe) ? In other words, if I just use those high level multi-threading APIs (e.g. ThreadPool/ThreadPoolExecutor) without messing around cuda stream APIs, it’ll be very hard to create some race conditions or unsafe situations?</p>
<p>Context: I’m testing IO efficiency with various settings (threadpool, processpool, dataloader workers, etc) and wanted to make sure my code does not have any obvious bugs (which can be a headache when mixed with concurrent + async + mixed CPU/GPU operations here and there).</p>