akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Spectrogram of signal normalized between -1 and 1 not the same as original signal

username: ndelamaide
cooked: <p>Hi,</p>
<p>For one of my models I’m using torchaudio.transforms.Spectrograms and realized that the computed spectrogram of the original signal and the one that was min-max normalized don’t match.</p>
<p>Here is the spectrogram of the normalized signal, as I can only upload one picture. There shouldn’t be only one band at the bottom.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68e1455fdee5b909a54e983b0ba7c532d2bfd2da.png" data-download-href="https://discuss.pytorch.org/uploads/default/68e1455fdee5b909a54e983b0ba7c532d2bfd2da" title="newplot (7)"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68e1455fdee5b909a54e983b0ba7c532d2bfd2da_2_690x339.png" alt="newplot (7)" data-base62-sha1="eXOisLqVr7LvacgtSoQiVyJY3N0" width="690" height="339" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68e1455fdee5b909a54e983b0ba7c532d2bfd2da_2_690x339.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68e1455fdee5b909a54e983b0ba7c532d2bfd2da_2_1035x508.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/8/68e1455fdee5b909a54e983b0ba7c532d2bfd2da_2_1380x678.png 2x" data-dominant-color="4741A1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">newplot (7)</span><span class="informations">1734×854 15.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I used torchaudio.transforms.Spectrogram(n_fft=256, hop_length=4, normalized=True, power=2) to compute the spectrograms. Moreover, the input data types are the same (float64).</p>
<p>Any ideas ?</p>
CUDA OOM when resume training

username: Di_Xiu
cooked: <p>Hi, all</p>
<p>I’m quite headache now.</p>
<p>I have used 32cards to train 40epochs. While the training stops at 30th epoch.</p>
<p>I wanna resume training from the checkpoint. Although I successfully load the checkpoint, the training script declares OOM at the first glance of minibatch.</p>
<p>Here is my saving and loading scripts.</p>
<pre><code class="lang-python">def save_model(args, epoch, model, model_without_ddp, optimizer, loss_scaler):
    output_dir = Path(args.output_dir)
    epoch_name = str(epoch)
    if loss_scaler is not None:
        checkpoint_paths = [output_dir / ('checkpoint-%s.pth' % epoch_name)]
        for checkpoint_path in checkpoint_paths:
            to_save = {
                'model': model_without_ddp.state_dict(),
                'optimizer': optimizer.state_dict(),
                'epoch': epoch,
                'scaler': loss_scaler.state_dict(),
                'args': args,
            }
            save_on_master(to_save, checkpoint_path)

def load_model(args, model_without_ddp, optimizer=None, loss_scaler=None):
    if args.resume:
        checkpoint = torch.load(args.resume, map_location="cpu")
        model_without_ddp.load_state_dict(checkpoint['model'])
        if 'optimizer' in checkpoint:
            if optimizer is not None:
                optimizer.load_state_dict(checkpoint['optimizer'])
</code></pre>
<p>Here is my minimal instance. Here, the optimizer is <a href="https://github.com/cybertronai" rel="noopener nofollow ugc">cybertronai</a>/<strong><a href="https://github.com/cybertronai/pytorch-lamb" rel="noopener nofollow ugc">pytorch-lamb</a></strong></p>
<pre><code class="lang-python">model = MyModelClass()
model.to(device)
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu], find_unused_parameters=False)
model_without_ddp = model.module

param_groups = optim_factory.add_weight_decay(model_without_ddp, args.weight_decay)
optimizer = Lamb(param_groups, lr=args.lr, betas=(0.9, 0.999), eps=1e-4)

load_model(args=args, model_without_ddp=model_without_ddp, optimizer=optimizer)
</code></pre>
<p>I oberserve the phenomenon that, when training from scratch, all works well. But after loading the checkpoint, the training fails at the first backward.</p>
<p>I found  [URL-hhttps://discuss.pytorch.org/t/gpu-memory-usage-increases-by-90-after-torch-load/9213/14] is related, but its method doesn’t work.</p>
<p>Any help will be quite appreciated.</p>
Are there any good tutorials on transfer learning for pytorchvideo?

username: aendrs
cooked: <p>Is there any good tutorial about doing transfer learning with pytorchvideo?<br>
Especifically I would like to do video recognition using the X3D model, leveraging a pretrained model and training it with my specific dataset (low number of classes).<br>
Thanks for pointing me out to the correct direction.</p>
Model's non-deterministic behaviour: Model produces different bounding box coordinates for the same image

username: kenhas
cooked: <p>Hi all,</p>
<p>I have a ResNet-50 model trained (using detectron2 framework) to detect just 1 object.<br>
The problem is for the same image the model will produce slightly different bounding box values:<br>
<span class="hashtag">#1</span></p>
<pre><code class="lang-auto">({'pred_boxes': Boxes(tensor([[832.2043, 473.2896, 986.7280, 521.5142]])),
  'scores': tensor([1.0000]),
  'pred_classes': tensor([0])},
 array([[0.99999404]], dtype=float32))
</code></pre>
<p><span class="hashtag">#2</span></p>
<pre><code class="lang-auto">({'pred_boxes': Boxes(tensor([[832.2321, 473.2645, 986.7278, 521.4969]])),
  'scores': tensor([1.0000]),
  'pred_classes': tensor([0])},
 array([[0.99999404]], dtype=float32))
</code></pre>
<p>I’d appreciate your guidance, as I’m very new to this.</p>

username: tom
cooked: <p>Can you describe what kind of problems this causes for you? This is essentially a relative error in the 1e-4/5 range, which I would not worry about unless I absolutely have to.</p>
<p>Best regards</p>
<p>Thomas</p>
Is it possible to force full precision for linear layer during prediction?

username: Anandh_Perumal_Konar
cooked: <p>In the pytorch docs, it is stated that:</p>
<p><code>torch.amp provides convenience methods for mixed precision, where some operations use the torch.float32 (float) datatype and other operations use lower precision floating point datatype (lower_precision_fp): torch.float16 (half) or torch.bfloat16. Some ops, like linear layers and convolutions, are much faster in lower_precision_fp. Other ops, like reductions, often require the dynamic range of float32. Mixed precision tries to match each op to its appropriate datatype.</code></p>
<p>Does that mean, linear layer will always be float16?<br>
I’m training a regression model, and final layer is a linear layer. My model predicts inf for value above 65k. Let me know, if I’m missing anything.</p>

username: tom
cooked: <p>You can wrap the evaluation of the last linear layer in an <a href="https://pytorch.org/docs/stable/amp.html#torch.autocast" rel="noopener nofollow ugc"><code>autocast(enabled=False)</code></a> block, it is briefly touched upon at the bottom of the description of the autocast context manager.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Anandh_Perumal_Konar
cooked: <p>Thanks for the input.<br>
Does that mean, linear layer will always be float16?<br>
I’m training a regression model, and final layer is a linear layer. My model predicts inf for value above 65k.</p>

username: tom
cooked: <p>By default, linear layers will use fp16 computation. The context manager allows you to deviate from that by turning off AMP for certain bits. For example, you could make a “LinearFP32” subclass of Linear by calling the super forward inside the with block mentioned above.</p>
Loss is NaN | vanishing/exploding gradients?

username: aizyuval
cooked: <p>Hello, Im trying to implement the custom dataset mentioned <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="noopener nofollow ugc">here</a>, into a model that can detect faces.</p>
<p><em>The problem:</em> while training, the loss is nan. I dont know what to do.</p>
<p><strong>Code:</strong></p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):

        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 3)
        self.conv2 = nn.Conv2d(6, 9, 3)
        self.fc1 = nn.Linear(9*171*171, 1500)
        self.fc2 = nn.Linear(1500, 544)
        self.fc3 = nn.Linear(544, 136)


    def forward(self, x): 
        x= F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x= torch.flatten(x,1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


transformer = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
])
batchsize = 4
trainset  = projectdata(csv_file = 'face_landmarks.csv', root_dir = 'faces',transform=transformer)
train_loader = DataLoader(trainset, batch_size=batchsize, shuffle=True, num_workers=0)

optimizer = optim.SGD(net.parameters(), lr=0.0001, momentum=0.9) 
def cross_entropy(input, target):
return torch.mean(-torch.sum(target * torch.log(input), 1))

for epochs in range(2):
  running_loss = 0.0
  for i,data in enumerate(train_loader,0):
    input,labels = data

    optimizer.zero_grad()
    outputs = net(input)#136 PREDICTED annotations tensor x4 (batch size)
    outputs = outputs.reshape((batchsize,68,2)) #reshaped to fit the labels shape

    loss = cross_entropy(outputs,labels) #calculating the loss
    loss.backward() 
    optimizer.step()


</code></pre>
<p>Ive tried using <code>leaky_relu</code>, but with no success either.<br>
Also tried to switch the optimizer to <code>Adam</code>.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>You custom <code>cross_entropy</code> function looks wrong, as it will apply <code>torch.log</code> on the model outputs directly, which are logits and can thus be negative as well as positive.<br>
Negative values will create <code>NaN</code>s which would then explain your invalid loss values.<br>
What’s the reason you are not using the correct and numerically stable <code>nn.CrossEntropyLoss</code> criterion?</p>

username: aizyuval
cooked: <p>Hi, thank you and sorry for the late response.<br>
I thought I needed to use a custom cross_entropy in order to handle with 2 arrays.</p>
<p>Anyway, I switched it into <code>nn.CrossEntropyLoss</code> but the loss is NaN again.<br>
It seems that the gradients often explode. sometimes loss is 27000, and then 50000, then NaN…</p>
How to use torchmetrics.FBetaScore with weights?

username: evilroach
cooked: <p>Sorry about posting my question here, because I can <strong>NOT</strong> found the forum of torchmetrics.<br>
If any one would pls tell me where is it, I’ll appreciate.</p>
<p>I need to use torchmetrics.FBetaScore with weights, but I can not understand the usage.<br>
I’m resolving a binary classification problem, and the positive class should be more considered when validating.</p>
<p>Even though the function can accept a parameter <strong>“average”</strong> with “<strong>weighted</strong>” as real value, I need more detail example to mimic, and there is no sample code about it everywhere.</p>
<p>According to the docs literally, it looks like that <code>weighted</code> is of string type, but the type that I want to pass should be two floats.</p>
<p>How to pass two weights into the function?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="evilroach" data-post="1" data-topic="162320">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/evilroach/48/53515_2.png" class="avatar"> evilroach:</div>
<blockquote>
<p>Even though the function can accept a parameter <strong>“average”</strong> with “<strong>weighted</strong>” as real value, I need more detail example to mimic, and there is no sample code about it everywhere.</p>
<p>According to the docs literally, it looks like that <code>weighted</code> is of string type, but the type that I want to pass should be two floats.</p>
</blockquote>
</aside>
<p>Would using <code>'none'</code> as the <code>average</code> argument work, as it would return the “raw” metric which you could then process in your custom weighting approach?</p>

username: evilroach
cooked: <p>Thank you for your help!<br>
I calculated them by myself, and found that they are same with the opponents in torchmetics.</p>
<p>The reason why I can’t understand the docs is that I mistake that the function accepts some weights as parameters.<br>
In fact, <strong>“FBetaScore”</strong> only accepts a parameter named “<strong>average</strong>”, that only can be a string value such as ‘none’/‘weights’ and so on.<br>
There is no opportunity we can specify some weights, they always accords to a fixed formula, if my comprehension is right.</p>
<p>Thanks!</p>
RuntimeError: "tanh" "_vml_cpu" not implemented for 'Half'

username: vivian528899
cooked: <p>Hi All,</p>
<p>There is no error when I trained my code on GPU and used the <code>dtype=torch.float16</code>.<br>
But when I run my code on CPU, I got the error</p>
<blockquote>
<p>RuntimeError: “tanh” “_vml_cpu” not implemented for ‘Half’</p>
</blockquote>
<p>I assume because <code>torch.half</code> isn’t supported by <code>torch.tanh</code> on CPU?<br>
Is there any method can do <code>tanh</code> in <code>dtype=torch.float16</code>?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Yes, you are right and the <code>float16</code> support on CPU is sparse as no speedups are expected, if I’m not mistaken. The default mixed-precision <code>dtype</code> on the CPU would be <code>bfloat16</code>.</p>
The CUDA kernel of torch.nn.CrossEntropyLoss fails when the input tensor is too large

username: Koramajin
cooked: <p>I got a problem with the CUDA kernel of torch.nn.CrossEntropyLoss. It fails in its forward function caused by an illegal memory access.  I posted my <a href="https://github.com/pytorch/pytorch/issues/85786" rel="noopener nofollow ugc">issue</a> in github. Hope it will get reponsed quickly.</p>
<p>For large input tensors currently, I divide the input tensor to multiple segments and call multiple times of F.cross_entropy to these segments. Is there any good way to apply cross entropy loss to large tensors?</p>

username: ptrblck
cooked: <p>Yes, GitHub issues are seen and it seems to be a duplicate of <a href="https://github.com/pytorch/pytorch/issues/85005" class="inline-onebox">CUDA Illegal memory access on CrossEntropyLoss with large batch size, cu113, torch 1.12.1 · Issue #85005 · pytorch/pytorch · GitHub</a></p>
Terminating with uncaught exception of ConvPackedParamsBase

username: Kareem_Shaik
cooked: <p>I am seeing this error in loading:<br>
<strong>terminating with uncaught exception of type c10::Error: Type c10::intrusive_ptr&lt;ConvPackedParamsBase&lt;2&gt;, c10::detail::intrusive_target_default_null_type&lt;ConvPackedParamsBase&lt;2&gt; &gt; &gt; could not be converted to any of the known types.<br>
Exception raised from call at /Users/distiller/project/aten/src/ATen/core/jit_type.h:1703</strong></p>
<p>TORCH_LIBRARY_IMPL(quantized, QuantizedCPU, m) {<br>
m.impl(TORCH_SELECTIVE_NAME(“quantized::conv1d”),          QConv1dInt8::run);<br>
}</p>

username: Kareem_Shaik
cooked: <p>any solution to this problem…</p>
Know image number during data load

username: cbd
cooked: <p>When “for i, data in enumerate(dataset):” called, dataloader call one image at time and do pre processing in “<strong>getitem</strong>” one image at time and create batch of 8 image in “data”. How can i track the number of the image in current process in “<strong>getitem</strong>”?</p>
<p>For example in first batch when first image process, i need number 1. When first image of next batch process, i need number 9 in “<strong>getitem</strong>” function.</p>

username: ptrblck
cooked: <p>If you are not shuffling the dataset, you could just use the <code>index</code> in <code>__getitem__</code>. However, I don’t know what exactly your use case is and how multiple workers and shuffling should be handled. You might be able to create a custom sampler, which could pass this additional information to the <code>__getitem__</code>.</p>

username: cbd
cooked: <p>Thanks. Passing the index helps.</p>
Process data of torch.half on CPU

username: mcorange
cooked: <p>Hi,<br>
I want to process (with method in torch.nn.functional) some data of <code>torch.half</code> type on CPU in fetching phase of data loader. I know one possible way is to convert the data to <code>torch.float</code> first  and then process it. Since I want to feed data of <code>torch.half</code> to my model, I have to convert the data back to <code>torch.half</code> again after processing.</p>
<p>Is there a neater way to do it?</p>
<p>Thanks in advance!</p>

username: ptrblck
cooked: <p>I assume you are running into some unsupported methods using <code>float16</code> on the CPU during your preprocessing phase and thus would need to transform the data to <code>float32</code>?<br>
If so, then note that your actual model might also run into the same issues as the default mixed-precision support on the CPU is for <code>bfloat16</code>, not <code>float16</code>.</p>

username: mcorange
cooked: <p>Yes, this is exactly what I ran into. However, my model is supposed to run on the GPU and I should have no concern about these methods right?</p>
<p>What I wanna do is to preprocess the <code>float16</code> data and keep the data type before moving to GPU for training my model with mixed-precision mode. Maybe there is a better way than <code>float16</code>(original data) → <code>float32</code>(for preprocessing on CPU) → <code>float16</code>(after preprocessing, converted back for GPU training)?</p>
RuntimeError: Error compiling objects for extension while installing detectron2

username: tomi
cooked: <p>I’m trying to install detectron2 on my local machine. A cpp_extension.py located in</p>
<p><code>C:\Users\Tomi\miniconda3\envs\pytorch_env\lib\site-packages\torch\utils\cpp_extension.py</code></p>
<p>fails to compile and throws a</p>
<pre><code class="lang-auto">raise RuntimeError(message) from e
    RuntimeError: Error compiling objects for extension
</code></pre>
<p>I just finished installing the tensorflow object detection API so I have pycocotools and other c++ build tools required. Are there more extensions of c++ tools I would have to install?</p>
<p>OS: windows<br>
Visual Studio tools installed: 2015 and 2019 build tools, windows 10 sdk, windows universal crt and crt sdk<br>
pytorch version: 1.12.1</p>
<p>command leading to error:<br>
<code>python -m pip install -e detectron2</code></p>
Error in cross entropy loss in last data loader iteration

username: bryan123
cooked: <pre><code> for e in range(args.epochs):
    models['backbone'].train()
    models['module'].train()
    train_loss = 0.0
    for batch_idx, (inputs, targets) in enumerate(trainloader):
        inputs, targets = inputs.to(device), targets.to(device)
        outputs, features = models['backbone'](inputs)
        print(outputs.shape)
        print(targets.shape)
        target_loss = criterion(outputs, targets)

        if args.epochs &gt; args.epochs_loss:
            # After args.epochs_loss, stop the gradient from the loss prediction module propagated to the target model.
            features[0] = features[0].detach()
            features[1] = features[1].detach()
            features[2] = features[2].detach()
            features[3] = features[3].detach()
            features[4] = features[4].detach()
</code></pre>
<p>At the end of the last iteration of the data loader, specifically at the loss = criterion(outputs, targets) line,  I received the following error “AttributeError: ‘tuple’ object has no attribute ‘log_softmax’”. Is there any way to solve this problem?</p>

username: lfsalasg
cooked: <p>Hi <a class="mention" href="/u/bryan123">@bryan123</a>, since you do not provide the definition of criterion, we cannot help you any further. However python is given you a hint: criterion returns a tuple, not a tensor. The softmax function can be applied only to tensors. Please check your criterion’s definition and the return statement</p>

username: bryan123
cooked: <p>Hi <a class="mention" href="/u/lfsalasg">@lfsalasg</a>, I have just resolved the problem. In the validation loader iterations, I only returned outputs (there should have been outputs and features) which caused the error.</p>
Custom dataset getitem return label as integer or tensor, return single data or a range of data?

username: Haizhuolaojisite
cooked: <p>The custom dataset will return image in tensor and its label. My questions are:</p>
<ol>
<li>What is the data format of label class? If return label as a tensor, which one is correct:</li>
</ol>
<pre><code class="lang-auto">class_id = torch.tensor(class_id) ---&gt;dataloader return label size of [batch]
or 
class_id = torch.tensor([class_id])---&gt;dataloader return label size of [batch, 1],here 1 is dimension of label
</code></pre>
<ol start="2">
<li>Can getitem method return a range of data points? I know dataset[0] return first element, but is dataset[2:10] feasible in custom dataset and dataloader? If feasible, how?</li>
</ol>
<p>Can anyone help me if possible please? Thanks a lot! I look forward to any reply!!</p>

username: eqy
cooked: <ol>
<li>I’m not sure it is expected that the label (if it is a class id) would be transformed to a tensor; see the canonical <code>ImageFolder</code> implementation here:<br>
<a href="https://pytorch.org/vision/main/_modules/torchvision/datasets/folder.html#ImageFolder" class="inline-onebox" rel="noopener nofollow ugc">torchvision.datasets.folder — Torchvision main documentation</a>
</li>
</ol>
<p>If I remember correctly <code>[batch]</code> should be fine and is expected for standard loss functions such as cross entropy loss: <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html" class="inline-onebox" rel="noopener nofollow ugc">CrossEntropyLoss — PyTorch 1.10 documentation</a> (see docs saying labels should have shape <code>(N)</code>).</p>
<ol start="2">
<li>Yes, potentially if <code>__getitem__</code> implemented slicing e.g., <a href="https://stackoverflow.com/questions/2936863/implementing-slicing-in-getitem" class="inline-onebox" rel="noopener nofollow ugc">python - Implementing slicing in __getitem__ - Stack Overflow</a>. But for common use cases slicing is not needed or desirable since the typical approach is to rely on dataloaders <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data — PyTorch 1.10 documentation</a> to automatically batch (and shuffle) your data.</li>
</ol>

username: Brando_Miranda
cooked: <p>this is what I’m doing. Isn’t it standard?</p>
<pre><code class="lang-auto">        # self.data, self.targets = self._load_data()
        del self.datasets
        self.target_transform = lambda data: torch.tensor(data, dtype=torch.int)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx: int) -&gt; tuple[Tensor, Tensor]:
        x = self.data[idx]
        y = self.indices_to_labels[idx]
        if self.target_transform is not None:
            y = self.target_transform(y)
        return x, y
</code></pre>
How to quantize the weight not the activation for QAT

username: Rami_Ismael
cooked: <p>I was wondering how to use the QCofig object to quantize the weight only. However, for the activation at all. The purpose has been shown the quantize the only weight decrease the size of the model, and the difference in performance 32 bit to 8-bit model is much smaller.</p>

username: jerryzh168
cooked: <p>you can use weight only fake quant qconfig: <a href="https://github.com/pytorch/pytorch/blob/master/torch/ao/quantization/qconfig.py#L201" class="inline-onebox" rel="noopener nofollow ugc">pytorch/qconfig.py at master · pytorch/pytorch · GitHub</a>, we don’t have kernel support for most of the weight only quantized ops though</p>
Bloom float16 quantization fail

username: Annie-Sihan-Chen
cooked: <p>I try to use quantize_fx to quantize huggingface bloom model to float16.<br>
I use exactly the same way as the torch quantization test/tutorial file. The code runs well except that when I try to print the result type, it still float32 type which means the quantization never happens.<br>
Here is the patch I use:</p><aside class="onebox githubgist" data-onebox-src="https://gist.github.com/AmosLewis/20811123b15bb5d818a3e8da545075c8">
  <header class="source">

      <a href="https://gist.github.com/AmosLewis/20811123b15bb5d818a3e8da545075c8" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/AmosLewis/20811123b15bb5d818a3e8da545075c8" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/AmosLewis/20811123b15bb5d818a3e8da545075c8</a></h4>

  <h5>bloom_fp16.py</h5>
  <pre><code class="Python">import torch
from transformers import AutoModelForSequenceClassification

class HuggingFaceLanguage(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.model = AutoModelForSequenceClassification.from_pretrained(
            "bigscience/bloom-560m",  # The pretrained model.
            num_labels=2,  # The number of output labels--2 for binary classification.
            output_attentions=False,  # Whether the model returns attentions weights.</code></pre>
    This file has been truncated. <a href="https://gist.github.com/AmosLewis/20811123b15bb5d818a3e8da545075c8" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: jerryzh168
cooked: <p>the result will stay in float32 by default, you will need to check the ops used in the model to see if the model is actually quantized or not, by printing the model itself.</p>
How to quantize a trained model to INT8 and run inference on GPU

username: SM19
cooked: <p>Hi,</p>
<p>I need to quantize my model to INT8 using either PTQ or QAT or both and finally run inference on GPU using tensorrt.</p>
<p>I have seen the static quantization page which says quantization is only available on CPU. Is it still the case? Is there any way to achieve this on GPU?</p>
<p>I have tried the pytorch-quantization toolkit from torch-tensorrt using fake quantization. However, after compiling the exported torchscript using torch.int8, my model size and inference speed are the same as that with FP16.</p>
<p>Please let me know if there is some example/ blog explaining this.</p>
<p>Best Regards<br>
Sambit</p>

username: jerryzh168
cooked: <aside class="quote no-group" data-username="SM19" data-post="1" data-topic="162196">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sm19/48/48889_2.png" class="avatar"> SM19:</div>
<blockquote>
<p>I have tried the pytorch-quantization toolkit from torch-tensorrt using fake quantization. However, after compiling the exported torchscript using torch.int8, my model size and inference speed are the same as that with FP16.</p>
</blockquote>
</aside>
<p>what are the ops in your model? I think tensorrt would give good speedup for a model with a lot of convs, but not sure for other ops.</p>
<p>pytorch-quantization from torch-tensorrt is a flow from NVidia. we also have a flow from pytorch quantization side, please take a look at tests in: <a href="https://github.com/pytorch/TensorRT/blob/master/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L392" rel="noopener nofollow ugc">https://github.com/pytorch/TensorRT/blob/master/py/torch_tensorrt/fx/test/quant/test_quant_trt.py#L392</a></p>
Relationship between interpolate and grid_sample

username: Wesley_Neill
cooked: <p>We have been using grid_sample at work to sample images (and other data types) between known values. This seems like the equivalent of upsampling.</p>
<p>I’ve tested that when I direct the grid sample to the scaled (x, y) locations of known values, I get back the known values. This is good.</p>
<p>For a new task, I need to actually upsample some data on a grid. There are associated spatial coordinates for each “pixel” value in our data. The horizontal and vertical delta between spatial coordinates is not equal, and we need it to be. It seems that <code>functional.interpolate</code> has the utility we need.</p>
<p>However, <code>interpolate</code> does not preserve the original values like <code>grid_sample</code> does. It decreases large values by a large percent (20-30%). This is undesirable for us.</p>
<p>One co-worker has suggested that we implement our own version of interpolation using grid_sample. I’m wondering</p>
<ul>
<li>about any pit falls to this idea</li>
<li>is the dilution of large values when using <code>interpolate</code> expected behavior?</li>
</ul>
Changing torch_extensions path

username: henrique
cooked: <p>Hi guys,</p>
<p>Is there a way to change the default extension path from ~/.cache/torch_extensions/ to something else?<br>
I couldn’t find anything on the documentation.</p>
<p>Thank you,</p>

username: ptrblck
cooked: <p>You should be able to set this path via <code>TORCH_EXTENSIONS_DIR</code>. If this env variable is not set, the (system) default will be used.</p>
Is there a way to specify which device transport to use when initializing?

username: gc-fu
cooked: <p>The situation is that our project is running on top of <a href="https://github.com/gramineproject/gramine" rel="noopener nofollow ugc">Gramine</a>, which only supports a subset of all the syscalls.</p>
<p>When I try to run PyTorch on top of Gramine, an error occurred, which basically caused by the use of socket with domain “AF_NETLINK”.   After investigation, I thought that it was caused by the use of RDMA in gloo::transport::ibverbs::Device.</p>
<p>The Gramine is a LibOS, and it does not support this domain socket.</p>
<p>At the same time, I notice that there are four different types of devices, namely:<br>
gloo::transport::ibverbs::Device<br>
gloo::transport::tcp::Device<br>
gloo::transport::tcp::tls::Device<br>
gloo::transport::uv::device</p>
<p>So is there a way to specify which device to use when running PyTorch with gloo backend?</p>
<p>I notice that PyTorch checks a environment variable called <strong>GLOO_DEVICE_TRANSPORT</strong>, is this the right way to specify which device to use in gloo?</p>

username: kwen2501
cooked: <p><a class="mention" href="/u/osalpekar">@osalpekar</a> I wonder if you have any suggestion? Thanks a lot!</p>
.to("cuda:1", non_blocking=True) creates `cudaStreamWaitEvent`

username: NouamaneTazi
cooked: <p>Hello everyone! When I try this script:</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    seed = 0
    torch.manual_seed(seed)
    torch.cuda.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)

    for _ in range(2):
        stream = torch.cuda.current_stream()

        x = torch.rand(32, 256, 220, 220).cuda()

        # t = (x.min() - x.max()).to(torch.device("cpu"), non_blocking=True)
        t = (x.min() - x.max()).to("cuda:1", non_blocking=True)

        print(stream.query()) # False - Checks if all the work submitted has been completed.
        print(t)
        stream.synchronize() # wait for stream to finish the work
        
        # time.sleep(2.)
        print(stream.query()) # True - work done
        print(t)
</code></pre>
<p>The <code>.to("cuda:1", non_blocking=True)</code> operation creates unexpected <code>cudaStreamWaitEvent</code> events which forces the synchronization between the two GPUs.</p>
<p>Do you know why is it different from CPU-GPU copy operations? And is there a way to avoid synchronizing the two GPUs when making copies?</p>
<p>ps: I also tried <code> t = (x.min() - x.max()).detach().to("cuda:1", non_blocking=True)</code>, it gives the same result</p>

username: ptrblck
cooked: <p><code>cudaStreamWaitEvent</code> is not synchronizing the GPU, but waits on an event to finish.<br>
In your use case, I guess <a href="https://github.com/pytorch/pytorch/blob/08820cb030927200c92733436f454f5fcf19c6bd/aten/src/ATen/native/cuda/Copy.cu#L72-L85">this code</a> is used:</p>
<pre><code class="lang-auto">  CUDAStream copy_stream = getCurrentCUDAStream(src_device.index());
  if (src_device != dst_device) {
    // This is a cross-device copy on the src current stream and dst current
    // stream. We perform a two-way barrier between both devices' streams
    // before the copy. This ensures that any write-after-write and
    // write-after-read dependencies on the destination side are handled, so
    // that no one is operating on the dst memory when we perform the copy.
    // src waits on dst barrier (src already waits on src)
    CUDAEvent dst_ready;
    device_guard.set_device(dst_device);
    dst_ready.record(getCurrentCUDAStream(dst_device.index()));

    device_guard.set_device(src_device);
    dst_ready.block(copy_stream);
  }
</code></pre>
<p>and <a href="https://github.com/pytorch/pytorch/blob/08820cb030927200c92733436f454f5fcf19c6bd/aten/src/ATen/native/cuda/Copy.cu#L114-L124">this</a>:</p>
<pre><code class="lang-auto">  if (src_device != dst_device) {
    // dst waits on src barrier (dst already waits on dst). We cannot
    // operate on dst's copy until the copy is complete.

    // Still on src_device, record stream event
    CUDAEvent src_ready;
    src_ready.record(copy_stream);

    device_guard.set_device(dst_device);
    src_ready.block(getCurrentCUDAStream(dst_device.index()));
  }
</code></pre>
<p>which is used to avoid write-after-write and write-after-read situations.</p>
Inf values from Conv2d

username: Sri2001
cooked: <p>I am training a segmentation model using unet. In the decoder part of network, The input to a Conv2d layer (which is self.conv[0] here) is x (the extracted features from a batch of imgs), and it doesn’t have nans or infs. I am not using bias in the Conv2d and none of the weights (or grads) in it is inf or nan. But I am still getting infs in the output of Conv2d layer. How is this possible?<br>
I am attaching the issue screenshot (from VS Code debug mode).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7126e2fbcb5754d97fa00d8e8a0322fddd99b18.png" alt="image" data-base62-sha1="wY9Mwxfh08xjQJan2jFmPrpxjkc" width="463" height="305"></p>

username: Sri2001
cooked: <p>Also the screenshot of the relevant part of model class is below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e77b1161ad9599df539666f885f644a64f6d72a8.png" data-download-href="https://discuss.pytorch.org/uploads/default/e77b1161ad9599df539666f885f644a64f6d72a8" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e77b1161ad9599df539666f885f644a64f6d72a8.png" alt="image" data-base62-sha1="x1LXYMN3YYPHtJs4oHiKvdHIVXa" width="690" height="255" data-dominant-color="252B2B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">819×303 31.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there something I am missing? This feels not specific to this problem or network, but to the Conv2D operation itself.</p>

username: ptrblck
cooked: <p>You might be running into an overflow during the conv calculation. Could you check the <code>min/max/mean</code> stats of the weights as well as the inputs?</p>

username: Sri2001
cooked: <p>Yeah, the max/min of input is within ±1000, and the weights are all between -1/1.</p>

username: ptrblck
cooked: <p>I don’t know how the channels are set, but given these ranges I wouldn’t expect to see any overflows for channel dims <code>&lt;=1024</code>. Could you share a minimal and executable code snippet which would reproduce the issue or any checkpoint?</p>
`unsupported datatype for constant'

username: Yten
cooked: <p>hi, I’m trying to test this code:</p>
<p><a href="https://pytorch-lightning.readthedocs.io/en/stable/notebooks/course_UvA-DL/13-contrastive-learning.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-lightning.readthedocs.io/en/stable/notebooks/course_UvA-DL/13-contrastive-learning.html</a></p>
<p>with mps device on M1 with torch version 1.13.0.dev20220913 but I get</p>
<p><code>/AppleInternal/Library/BuildRoots/a0876c02-1788-11ed-b9c4-96898e02b808/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm:343: failed assertion 'unsupported datatype for constant'</code></p>
<p>I have confirmed that mps device is available and works for other code</p>
Masking out padding for LSTM

username: sesale
cooked: <p>I have sequences which I padded to a fixed length (365 days) by inserting zeros at the missing time steps (so the padding is contained at varying time steps within the sequences). I then feed the sequences into an LSTM network in order to classify them.<br>
I created a mask which contains <code>True</code> if the value is 0 (padding) and <code>False</code> if not, s.t. the model does not take into account the zeros (I checked the masks and there are indeed values that are <code>False</code>, meaning no padding and the timestep should be taken into account by the model).<br>
However, for some of my sequences, the output of the backbone (before applying the linear layer for classification) results in <strong>only</strong> <code>nan</code> values.<br>
Does someone know why? Am I doing anything wrong?<br>
Is it correct to apply the masking before feeding the values through the lstm layer? I also tried to apply the masking afterwards. In that case, <code>out</code> contains float values but once the masking is applied, all values are <code>-inf</code>.</p>
<pre><code class="lang-auto">def forward(self, x, device):
        mask = x[:, :, 0].eq(0).unsqueeze(-1)
        mask = mask.to(device) # [batch_size, seq_len, 1] = [16, 365, 1]

        # masking out padded time steps
        x = x.masked_fill(mask.bool(), -np.inf) # now only some values are -inf as expected

        x = x.float() # [batch_size, seq_len, channels] = [16, 365, 3]

        # cell states
        h0 = (
            torch.zeros(layer_dim,  x.size(0), hidden_dim)
            .requires_grad_()
            .to(device)
        )
        # Initialize cell state
        c0 = (
            torch.zeros(layer_dim,  x.size(0), hidden_dim)
            .requires_grad_()
            .to(device)
        )
        # [batch_size, seq_len, hidden_dim] = [16, 365, 150]
        out, _ = self._lstm_layer(x, (h0.detach(), c0.detach()))  # now all values are nan
</code></pre>
Is there a function can perturb a tensor?

username: evilroach
cooked: <p>I want to perturb a column of the input feature data randomly, but the input data has been transformed into a torch.Tensor, I don’t want to transform it back into numpy.ndarray to do the perturbation.</p>
<p>Is there a existed function, can do this?</p>

username: srishti-git1110
cooked: <p>Hi,<br>
Please post the numpy code you need a pytorch equivalent for.</p>
NaN in Loss Function after 13 epochs

username: ThiruRJST
cooked: <p>I’m training EfficientNetV2_B3 model with my data, I have tried so many times but after 13 epochs, the loss becomes NaN. Until the 13th epoch, the model was training and validating good.</p>
<p>Dataset: Custom<br>
Model: EfficientNetV2_B3<br>
Loss Function: nn.CrossEntropyLoss<br>
Batch size: 32<br>
Training Mode: Mixed Precision<br>
LR Scheduler: ReduceLROnPlateau - Default settings<br>
Optimizer: Adam<br>
Initial Learning rate: 0.01</p>
<p>From 14th epoch to 17th epoch, both the training and validation loss remains NaN. can anyone could help me with this?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f21b101f8e68d5325c622db8c37787a7862a1d83.png" data-download-href="https://discuss.pytorch.org/uploads/default/f21b101f8e68d5325c622db8c37787a7862a1d83" title="Screenshot from 2022-07-24 12-20-55"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f21b101f8e68d5325c622db8c37787a7862a1d83_2_690x496.png" alt="Screenshot from 2022-07-24 12-20-55" data-base62-sha1="yxLwRooZOvtelrKLlhtqg70gkzp" width="690" height="496" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f21b101f8e68d5325c622db8c37787a7862a1d83_2_690x496.png, https://discuss.pytorch.org/uploads/default/original/3X/f/2/f21b101f8e68d5325c622db8c37787a7862a1d83.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/2/f21b101f8e68d5325c622db8c37787a7862a1d83.png 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-07-24 12-20-55</span><span class="informations">1033×744 28.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks in advance,<br>
Thiru</p>

username: fpozzi
cooked: <p>Hi <a class="mention" href="/u/thirurjst">@ThiruRJST</a>,</p>
<p>first of all, I would check the input and the output (of the model) to see if they are originating the NaN values.</p>

username: ThiruRJST
cooked: <p>okay are you asking me to check the dataset with current setup of augmentations?</p>

username: ThiruRJST
cooked: <p>what should I do if its output?</p>

username: fpozzi
cooked: <blockquote>
<p>okay are you asking me to check the dataset with current setup of augmentations?</p>
</blockquote>
<p>Look at the batch that produces NaN values, inspecting the input and the output of the model: maybe the loss value is NaN because the prediction contains some NaN values.</p>

username: ThiruRJST
cooked: <p><a class="mention" href="/u/fpozzi">@fpozzi</a> I tried as you instructed, the inputs weren’t NaN, only the outputs of the model was Nan and that’s the reason for loss becoming NaN. But I couldn’t find which layer of the model rises this issue.</p>
Model don't finish downloading

username: Abdelkareem_Elkhateb
cooked: <p>every time I try to download a pretrained model the download finish but the cell is still busy<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48da2eff7ffca86506e10352cdb4ae404cd542ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/48da2eff7ffca86506e10352cdb4ae404cd542ed" title="Screenshot from 2022-09-27 01-34-59"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48da2eff7ffca86506e10352cdb4ae404cd542ed_2_690x134.png" alt="Screenshot from 2022-09-27 01-34-59" data-base62-sha1="aotPI5oJJBm8hLcqvxglUTeq5tz" width="690" height="134" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48da2eff7ffca86506e10352cdb4ae404cd542ed_2_690x134.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48da2eff7ffca86506e10352cdb4ae404cd542ed_2_1035x201.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48da2eff7ffca86506e10352cdb4ae404cd542ed_2_1380x268.png 2x" data-dominant-color="373743"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-09-27 01-34-59</span><span class="informations">1920×373 85.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I am using jupyter notebook in vs code</p>
/torchvision/transforms/functional_pil deprecationwarning

username: Sam_Lerman
cooked: <p>I get this depreciation warning at some point from within torchvision. I’m not sure the exact line that triggered it.</p>
<pre><code class="lang-auto">/Users/samlerman/code/miniforge3/envs/ML/lib/python3.8/site-packages/torchvision/transforms/functional_pil.py:228: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  interpolation: int = Image.BILINEAR,
/Users/samlerman/code/miniforge3/envs/ML/lib/python3.8/site-packages/torchvision/transforms/functional_pil.py:295: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  interpolation: int = Image.NEAREST,
/Users/samlerman/code/miniforge3/envs/ML/lib/python3.8/site-packages/torchvision/transforms/functional_pil.py:311: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
  interpolation: int = Image.NEAREST,
/Users/samlerman/code/miniforge3/envs/ML/lib/python3.8/site-packages/torchvision/transforms/functional_pil.py:328: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
  interpolation: int = Image.BICUBIC,
</code></pre>
Question about vmap many models to many inputs

username: ryoukai
cooked: <p>Looking at the vmap documentation and just playing around with the function, it looks like you can map many models to many inputs as long as num_models == num_inputs i.e. the batch sizes are the same. My question is, what if there are N*M inputs, where N is the number of models. Say, for example, I have 10 models and 20 inputs. Can I vmap the models to the inputs such that each model does forward pass on two of the inputs?</p>
Segfault when looking up in CUDA pointer

username: Jegp
cooked: <p>I’m extracting a pointer for the underlying memory on the GPU for a tensor with bytes like so:</p>
<pre><code>auto opt = options_buffer = torch::TensorOptions()
                   .dtype(torch::kUInt8)
                   .device(torch::DeviceType::CUDA)
                   .memory_format(c10::MemoryFormat::Contiguous);
auto buffer = torch::zeros(size, opt);
auto *array = buffer-&gt;data_ptr&lt;uint8_t&gt;();
</code></pre>
<p>However, if I try to look up in that or do anything related to that pointer, it segfaults. But. Why? The memory should be contiguous. I’m imagining it has something to do with memory formatting and/or locks/access, but I can’t seem to find good documentation for it. I would also be grateful if you happen to know any resources that can help in this direction.</p>
<pre><code>array[0] // segfaults
</code></pre>

username: ptrblck
cooked: <p>It seems you are trying to access device data from the host, which is UB and could segfault.</p>

username: Jegp
cooked: <p>Thanks for the reply! I am indeed accessing it from the host. So, you would say this kind of access <strong>has</strong> to happen from a CUDA block?</p>
<p>The weird thing is that this worked in a previous installation. Did PyTorch add memory protection recently?</p>

username: ptrblck
cooked: <p>Yes, you need to either access the device array in a kernel or would need to copy it back to the CPU.<br>
I don’t think this should have ever worked as it’s expected behavior in CUDA. PyTorch did not add any memory protection etc.</p>
<p>E.g. look at <a href="https://gist.github.com/dpiponi/1502434">this simple example</a> which:</p>
<ul>
<li>allocates host and device memory</li>
<li>fills the host array with values</li>
<li>copies the host array to the device array via <code>cudaMemcpy</code>
</li>
<li>launches the compute kernel, which indexes the device array</li>
<li>copies the device array back to the host</li>
<li>prints the host array via indexing it</li>
<li>frees the allocations</li>
</ul>
<p>Now, add an invalid access e.g. via <code>printf("%d\n", da[0]);</code> in line 69 and you will get a Segmentation fault.</p>

username: Jegp
cooked: <p>Thank you for your replies! I can see why this <em>shouldn’t</em> indeed work. I resolved it by writing CUDA kernels to deal with the memory allocated on the GPU.</p>