akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Understanding "mask" dtype from TransformerEncoder forward

TeDataPro: <p>Dear all,</p>
<p>I am working with TransformerEncoder module. I understood the “mask” parameter from the forward function can be a boolean tensor where <strong>True</strong> indicates that it is forbidden to attend the token and <strong>False</strong> is the inverse.</p>
<p>In my case, I am working with sequence-to-sequece mask. Initially, I was converting my mask to float with the following code</p>
<pre><code class="lang-auto">mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
</code></pre>
<p>However, I realized it takes lots of time (~0.60 seconds) when length of context is &gt; 50. So I removed the float conversion and kept boolean values and it takes only ~0.06 seconds for equivalent context length.</p>
<p>Note that my masks shape are (batch_size, T, T) because each sample from the batch has have different mask.</p>
<p>My question is : what is the key difference between a boolean and float mask ? Do they give the same results ?</p>
<p>Thanks !</p>

ptrblck: <p>You might run into <a href="https://discuss.pytorch.org/t/floating-point-masks-are-not-supported-for-fast-path/205721">this limitation</a> disallowing the fast path if floating point masks are used.</p>
How to compare compute time and batch size between serial and parallel GPU runs

Jc112358: <p>Hi,<br>
I have a question about comparing wall times between parallel and serial GPU runs. My code uses DistributedDataParallel for GPU parallelization. I want to compare the wall time between using a single GPU and using multiple GPUs, but I’m not really sure how to handle the wall times with batching for each scenario. For example, if the serial GPU run uses a batch size of 10, and the DDP run on 2 GPUs uses a batch size of 10 (but split into 2 batches of 5 on each GPU), then can the wall times from each training routine be directly compared? I guess my confusion comes down to whether or not I should use a batch size of 20 when running on 2 GPUS (split into 2 batches of 10 on each GPU), or a batch size of 10 (split into 5 on each GPU), to compare with a serial run using a batch size of 10. Hopefully this question makes sense, and I’m happy to answer any questions.</p>

ptrblck: <p>Usually you would scale the global batch size with the number of GPUs. I.e. if you double the number of GPUs you would also double the global batch size, which corresponds to the same local batch size.</p>
Computational graph while extracting token embeddings

Debodeep_Banerjee: <p>Hi,<br>
I am an image captioning model that involves an underlying performance as well. Let me represent a pseudo version of my code.</p>
<p>model = image_caption_model()<br>
tokens, logprobs = model(input)<br>
ce_loss = criterion(gt_tokens, preds_tokens)<br>
embedding = model.embeddings.weight (the embedding weights)<br>
token_embeddings = embeddings(tokens)<br>
output2= model2(token_embeddings)<br>
combined_loss = output2+ce_loss<br>
combined_loss.backward()<br>
now, usually, I should not be using this as the the tokens are not differentiable. Hence, extracting the embeddings from the token may cause trouble for the backdrop. but when I am printing required_grad, I see  token_embeddings.require_grad is True. As the tokens are not differentiable, I assumed it to be False. Can someone explain me the reason and whether the current strategy is correct?</p>
MobileNetV2SSD in Pytorch

Cyrus1: <p>I am working on a MobileNetV2 with SSD head for object detection in ipynb, VSCode. My code has a class for the dataset (which is in YOLO format with image size 640x640), a block for the model (class for Bottleneck, MobileNetV2, SSDHead, MobileNetV2SSD), function defined for iou and anchor matching, function defined for ssd_loss, function defined for custom_collate_fn, and training loop.</p>
<p>When I run the code, I get the following error. How do I fix it… I need simple explanation as I am beginner in the field.</p>
<p>File c:\Users\ca8mc\Desktop\netv2v8\netv2v8_env\lib\site-packages\torch\nn\functional.py:3104, in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction, label_smoothing)<br>
[3102] if size_average is not None or reduce is not None:<br>
[3103] = _Reduction.legacy_get_string(size_average, reduce)<br>
 → [3104] return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)</p>
<p>ValueError: Expected input batch_size (4704) to match target batch_size (64).</p>

ptrblck: <p>Based on the error message you are using <code>nn.CrossEntropyLoss</code> which expects a model output and target with the same batch size (in <code>dim0</code>). Check why the batch sizes differ in your case and if you accidentally reshaped one tensor on a wrong way.</p>
Stabilizing a formula involving powers

sherlock.h: <p>Hello! I am trying to learn the alpha and beta in the following equation, but I am really having a hard time getting something that doesn’t throw <code>nan</code> during backward.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/1/1152530046d6fe58c2825577c42f6739db69d1e5.png" alt="image" data-base62-sha1="2teuJYCMmNYk0GKjBRmSJIuGg1n" width="606" height="254"></p>
<p>The code in question:</p>
<pre data-code-wrap="python"><code class="lang-python">(
    self.beta.abs()
    .mul(log(step / (self.n_steps - 1) + 1e-5))
    .exp()
    .mul(-1)
    .add(1)
    .log()
    .mul(self.alpha.abs())
    .exp()
    .mul(self.initial_lr.abs())
)
</code></pre>
<p><code>initial_lr</code>, <code>alpha</code>, and <code>beta</code> are <code>nn.Parameter</code>.</p>
<p>Any suggestions?</p>
<p>I’d also welcome some other interpolation recommendations with a nice concave to convex shape between (0, eps) and (T, 0). Though the boundary conditions must be satisfied.</p>
1% GPU Utilisation with 2 TB Dataset

Chandler_Kenworthy: <p>I have a very large dataset (&gt;2 TB raw) consisting of over 20 million images (44 x 44 with 10 channels). At the moment I convert my raw data to a HDF5 format with LZF compression. I then use PyTorch data loader when training my model but, after profiling I see my data loader takes 99.8% of the time during 1 epoch of training (1 epoch takes about 15 hrs currently on an A100).</p>
<p>Is there something I am doing wrong, can I construct my dataloader/dataset in a better way?</p>
<pre><code class="lang-auto">class PionDataset(Dataset):
    def __init__(self, hdf5_file, transform=None):
        self.hdf5_file = hdf5_file
        self.file = h5py.File(self.hdf5_file, 'r')
        self.dataset_length = self.file['EnergyLKr'].shape[0]
        self.labels = self.file['labels'][:]
        
        # Optional: apply transforms
        self.transform = transform
    
    def __len__(self):
        return self.dataset_length
    
    def get_target_frequencies(self):
        return torch.bincount(torch.tensor(self.file['labels'], dtype=torch.long))
    
    def __getitem__(self, idx):
        momentum = np.float32(self.file['momentum'][idx]) # MeV by default
        is_edge = torch.tensor(self.file['IsEdgeCluster'][idx], dtype=torch.float32)
        category = torch.tensor(self.file['labels'][idx], dtype=torch.long)
        
        # Read the image from the branch (44 x 44 x 10)
        image = torch.tensor(self.file['EnergyLKr'][idx], dtype=torch.float32) / momentum
             
        if self.transform:
            image = self.transform(image)
        
        # Return the image and labels as a tuple
        return image, is_edge, category
    
    def close(self):
        self.file.close()

transform = transforms.Compose([
    v2.RandomVerticalFlip(p=0.5)
])

# Create the dataset
hdf5_file = "/path/to/my/train.hdf5"
dataset = PionDataset(hdf5_file, transform=transform)

train_loader = DataLoader(
    dataset, 
    batch_size=1024,
    shuffle=True, 
    num_workers=2,
    prefetch_factor=1, 
    pin_memory=True,
    persistent_workers=True
)

# ...do model training

</code></pre>
<p>I noticed that using num_workers &gt; 0 slows the data loader even more! Doing something simple even like this:</p>
<pre><code class="lang-auto">batch = next(iter(train_loader))
</code></pre>
<p>takes more than 20 seconds.</p>
TorchFix release 0.6 is available

kit1980: <p>TorchFix is a linter for PyTorch-using code with autofix support, and this release brings new rules and other improvements.</p>
<p>See <a href="https://pypi.org/project/TorchFix/0.6.0/" rel="noopener nofollow ugc">https://pypi.org/project/TorchFix/0.6.0/</a> and <a href="https://github.com/pytorch-labs/torchfix" rel="noopener nofollow ugc">https://github.com/pytorch-labs/torchfix</a></p>

kit1980: <p>Couple of simple but interesting rules in this release: finding patterns like <code>torch.log(1 + x)</code> and <code>torch.exp(x) - 1</code> and suggesting replacing with more numerically stable equivalents.</p>
VAE is just predicting the mode of each input channel

Derek_Walkama: <p>Hello, I have a set of ~1300 binary features that I’m trying to embed using a fully-connected VAE. I noticed that the model, regardless of normalization, dropout, activation functions, etc, is no longer learning once it hits a prediction accuracy of 76%. I checked, and this value corresponds to predicting each feature using its mode.</p>
<p>I’ve even expanded the size (and latent space) of this model to check that there is enough complexity. I would expect a model like this to perfectly fit.</p>
<pre><code class="lang-auto">FcVae(
  (encode_list): ModuleList(
    (0-2): 3 x FCBlock(
      (fc_block): Sequential(
        (0): Linear(in_features=1282, out_features=1282, bias=True)
        (1): Identity()
        (2): LeakyReLU(negative_slope=0.02, inplace=True)
      )
    )
  )
  (encode_fc_mean): FCBlock(
    (fc_block): Sequential(
      (0): Linear(in_features=1282, out_features=1282, bias=True)
    )
  )
  (encode_fc_log_var): FCBlock(
    (fc_block): Sequential(
      (0): Linear(in_features=1282, out_features=1282, bias=True)
    )
  )
  (decode_list): ModuleList(
    (0-2): 3 x FCBlock(
      (fc_block): Sequential(
        (0): Linear(in_features=1282, out_features=1282, bias=True)
        (1): Identity()
        (2): LeakyReLU(negative_slope=0.02, inplace=True)
      )
    )
    (3): FCBlock(
      (fc_block): Sequential(
        (0): Linear(in_features=1282, out_features=1282, bias=True)
      )
    )
  )
)
[Network Embed] Total number of parameters : 14.803 M
</code></pre>
<p>I should mention the loss I’m using is BCE with ADAM optimizer.</p>

Derek_Walkama: <p>I think I figured it out, I was going a bit too hard with the regularization (weight_decay was 0.01). When I dropped weight_decay to 0.0001, it started to recover in-feature variance.</p>
The relationship between CUDA stream and thread

Aakira: <p>Hello everyone, I am quite confused about CUDA stream and thread. Here is my understanding:</p>
<ul>
<li>Thread: a processing thread, or can be understood as a kernel invocation, where I can launch many threads in parallel to perform the same task.</li>
<li>CUDA stream: from what I have read, it seems that CUDA stream is mainly used to optimize memcpyh2d → kernel call → memcpyd2h operations.</li>
</ul>
<p>So, if I understand correctly, CUDA stream and thread are meant to work together, (the code below is just pseudo code) instead of:</p>
<pre data-code-wrap="python"><code class="lang-python">def some_kernel(job):
   memcpyh2d(job)
   kernel(job)
   memcpyd2h(job)
</code></pre>
<p>we should use :</p>
<pre data-code-wrap="python"><code class="lang-python">def some_kernel(job):
   pieces = partion_into_pieces(job)
   for idx, p in enumerate(pieces):
      memcpyh2d(p, stream=idx)
      kernel(p, stream=idx) 
      memcpyd2h(p, stream=idx)
</code></pre>
<p>Am I understand correctly?</p>

ptrblck: <blockquote>
<p>A CUDA stream is a queue of GPU operations that are executed in a specific order. The order in which the tasks are added to this queue determines their order of execution.</p>
</blockquote>
<p>You can use multiple streams to overlap operations as described also <a href="https://discuss.pytorch.org/t/cuda-stream-not-working/155749/4">here</a>.</p>
<p>Multiple threads will execute the parallel portion of your CUDA kernel and <a href="https://developer.nvidia.com/blog/cuda-refresher-cuda-programming-model/">this blog post</a> might be a good starter.</p>
Which activation function should I use in the last layer output while using Resnet50?

Mert_Oguzhan: <p>Hi, I hope you are great.<br>
I have been creating my own cnn model with pytorch. But I am not sure which activation function should I use in resnet50 for the last layer (output) and what is the difference between them. Do you have any idea about that?</p>

Soumya_Kundu: <p>What is your task? Generally for image classification its softmax.</p>

ptrblck: <p>Keep in mind that commonly used loss functions for classification tasks require logits, i.e. no final activation function, if <code>nn.CrossEntropyLoss</code> is used, or log-probabilities, i.e. <code>F.log_softmax</code> as the final activation function, if <code>nn.NLLLoss</code> is used.</p>

Mert_Oguzhan: <p>I got glass images. I want to predict which glass image is good (I mean quality) which image is not good (1 or 0).</p>

deepsthecoder: <p>Your problem statement sounds like its a binary classification. Usually Sigmoid is preferred.<br>
While inference any value greater than equal to 0.5 is considered as 1 and lesser than 0.5 considered as 0.</p>
<p>For loss you can use Binary Cross Entropy<br>
<code>loss = F.binary_cross_entropy(torch.sigmoid(input), target)</code></p>
Margin Ranking loss in the wild

rhsimplex: <p>Hi all,</p>
<p>I’m a big fan of margin ranking loss for regression-ish problems. I couldn’t find any writeups on torch’s MarginRegressionLoss function outside of the context of contrastive embeddings, so I wrote my own:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://rhsimplex.github.io/2022/06/10/margin-ranking/">
  <header class="source">
      <img src="https://rhsimplex.github.io/images/favicon.ico" class="site-icon" width="48" height="48">

      <a href="https://rhsimplex.github.io/2022/06/10/margin-ranking/" target="_blank" rel="noopener nofollow ugc" title="09:29AM - 10 June 2022">R Henderson – 10 Jun 22</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:136/105;"><img src="https://rhsimplex.github.io/images/wine_quality_hist.svg" class="thumbnail" width="136" height="105"></div>

<h3><a href="https://rhsimplex.github.io/2022/06/10/margin-ranking/" target="_blank" rel="noopener nofollow ugc">A Neural Network Trick for Ranking Problems</a></h3>

  <p>Classification vs. Regression: the ultimate dichotomy in machine learning? It’s often presented this way, but I’ve found that many real-world problems are not so clear-cut. Often you want to predict t</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Posting here in the hopes it helps someone out.</p>
<p>Thanks!</p>

ptrblck: <p>Thanks for sharing the post! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ashukla_040: <p>This is so useful, thanks a lot!</p>
Speedup keypoint RCNN Inference

jay5: <p>I have a custom keypoint detection framework which I train using a few thousand coco annotated samples of everyday object images. The model works pretty well in predicting both keypoints and the bounding boxes, the training takes just a few minutes, but the inference is quite slow. The trained model which I save is about 250MB, (if I use model.half() for fp16 accuracy, it is about 130MB)</p>
<p>The train and load code is pretty straight forward.</p>
<pre><code class="lang-auto">anchor_generator = AnchorGenerator(
    sizes=(32, 64, 128, 256, 512),
    aspect_ratios=(0.25, 0.5, 0.75, 1.0, 2.0, 3.0, 4.0)
)

# Create the keypoint RCNN model
model = torchvision.models.detection.keypointrcnn_resnet50_fpn(
    weights=None,
    weights_backbone=ResNet50_Weights.DEFAULT,
    num_keypoints=20,
    num_classes=4,  # Background is the first class, object is the second class
    rpn_anchor_generator=anchor_generator
)

 # save model
 # load model from path later for inference

  # model = # load from path
  # Load pre-trained weights if available
  state_dict = torch.load(weights_path)
  model.load_state_dict(state_dict)
</code></pre>
<p>The tensor for which I want to perform inference is of [512 x 3 x 128 x 128], I.e. 512 RGB images of 128x128 pixels. I am batching like below to avoid Mem issues on my RTX4090.</p>
<pre><code class="lang-auto">
  with torch.no_grad():
       model.to(device)
       model.eval()
   
   rcnn_batch_size = 16

    # Loop over the list and perform predictions in batches
    with torch.no_grad():
        predictions = []
        for i in range(0, len(side_cam_rgb_tensors), rcnn_batch_size):
            batch_rgb_tensors = side_cam_rgb_tensors[i:i + rcnn_batch_size]
            batch_rgb_tensors = torch.stack(batch_rgb_tensors).to(device)
            batch_predictions = model(batch_rgb_tensors) # This is the slow statement that takes about 30 sec.

            .........

    for p in predictions:
	        # Take the p['scores'], p['keypoints'] and transform it appropriately

</code></pre>
<p>I was wondering if there are any steps I can do to speedup the inference process, perhaps with some trade off for accuracy or training time. Changing precision of model/input to fp16 improves the speed by about 10%. I have also tried to do torch.compile(model), which however results in an issue similar to this <a href="https://discuss.pytorch.org/t/compilation-hangs-too-many-threads/105933" class="inline-onebox">Compilation hangs: too many threads</a>. Any suggestion is appreciated.  Thanks</p>

MasihZoharian: <p>Dear Jay,</p>
<p>Have you found any solution for the issue?</p>
<p>Best Regards<br>
Masih</p>
Which method is best for Image Downsampling?

jawi289p: <p>I have a dataset with images sized 650x1250, and I want to downsample them for use with a deep learning model. The images contain very small objects, and resizing them to 320x320 has resulted in the model not learning these small features effectively.</p>
<p>Can you suggest any other methods to achieve a 320x320 image size while preserving small details?</p>

jawi289p: <p>what do you suggest <a class="mention" href="/u/ptrblck">@ptrblck</a>?</p>

ptrblck: <p>Would random crops work for your use case? This would keep the resolution while processing smaller parts of the model.</p>

jawi289p: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I couldn’t understand how random cropping will work because for UNet image dimension should be divisible by 32.<br>
I am using the original image with dimension 650x1250 and used random cropping transformation with probability 0.5 but the model is giving an error.</p>

ptrblck: <p>I don’t know what kind of error you are getting, but the random crop size can be set to a multiple of 32.</p>
Missing weight_fake_quant with prepare() vs prepare_qat()

shu_bekk: <p>Hi</p>
<p>I want to understand how quantization parameters are stored in PyTorch.<br>
Consider the following toy example:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

torch.random.manual_seed(0)

# Toy model. Two Linear layers &amp; a ReLU
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.a = nn.Linear(1, 8)
        self.act = nn.ReLU()
        self.b = nn.Linear(8, 2)
        self.quant = torch.ao.quantization.QuantStub()
        self.dequant = torch.ao.quantization.DeQuantStub()

    def forward(self, x):
        x = self.quant(x)

        x = self.a(x)
        x = self.act(x)
        x = self.b(x)

        x = self.dequant(x)
        return x

# Create original (unprepared model)
m_orig = Model()
print('Original model', m_orig)

# Create prepared model
m_orig.qconfig = torch.ao.quantization.get_default_qat_qconfig()
m = torch.ao.quantization.prepare(m_orig, inplace=False)
print('Prepared', m)

# Convert to quantized model
qm = torch.ao.quantization.convert(m, inplace=False)

</code></pre>
<p>In this case, the <code>weight_fake_quant</code> keys are missing from the state dict.</p>
<pre><code class="lang-auto">...
  (a): Linear(
    in_features=1, out_features=8, bias=True
    (activation_post_process): FusedMovingAvgObsFakeQuantize(
      fake_quant_enabled=tensor([1]), observer_enabled=tensor([1]), scale=tensor([1.]), zero_point=tensor([0], dtype=torch.int32), dtype=torch.quint8, quant_min=0, quant_max=127, qscheme=torch.per_tensor_affine, reduce_range=True
      (activation_post_process): MovingAverageMinMaxObserver(min_val=inf, max_val=-inf)
    )
  )
...
</code></pre>
<p>However, if I replace <code>the prepare()</code> with <code>prepare_qat()</code>, then these keys reappear</p>
<pre><code class="lang-auto">...
  (a): Linear(
    in_features=1, out_features=8, bias=True
    (weight_fake_quant): FusedMovingAvgObsFakeQuantize(
      fake_quant_enabled=tensor([1]), observer_enabled=tensor([1]), scale=tensor([1.]), zero_point=tensor([0], dtype=torch.int32), dtype=torch.qint8, quant_min=-128, quant_max=127, qscheme=torch.per_channel_symmetric, reduce_range=False
      (activation_post_process): MovingAveragePerChannelMinMaxObserver(min_val=tensor([]), max_val=tensor([]))
    )
    (activation_post_process): FusedMovingAvgObsFakeQuantize(
      fake_quant_enabled=tensor([1]), observer_enabled=tensor([1]), scale=tensor([1.]), zero_point=tensor([0], dtype=torch.int32), dtype=torch.quint8, quant_min=0, quant_max=127, qscheme=torch.per_tensor_affine, reduce_range=True
      (activation_post_process): MovingAverageMinMaxObserver(min_val=inf, max_val=-inf)
    )
  )
...
</code></pre>
<p>This is strange behavior to me. I would expect both <code>prepare()</code> and <code>prepare_qat()</code> to add weight quantization, but this isn’t the case. I guess I’m still trying to understand the differences between these two functions. Why wouldn’t <code>prepare()</code> also quantize the weights? Which one should I be using if I want to obtain quantized weights?</p>
Determining data flow in profiler generated chrome trace

Harsha_dokula: <p>I am currently working on <code>tensor-aware energy accounting</code>: <a href="https://dl.acm.org/doi/10.1145/3597503.3639156" rel="noopener nofollow ugc">https://dl.acm.org/doi/10.1145/3597503.3639156</a> for machine learning models. The goal is to attribute energy consumption to specific tensors within the model.</p>
<p>I was able to generate the trace , able to get the forward, backward flow segments , flow to GPU but not able to determine which events belong which tensor operations.<br>
For example, Gvien</p>
<pre data-code-wrap="chrome"><code class="lang-chrome">...
{
    "ph": "X", "cat": "cpu_op", "name": "autograd::engine::evaluate_function: ViewBackward0", "pid": 112835, "tid": 112835,
    "ts": 5284315312688.329, "dur": 21.660,
    "args": {
      "External id": 885,"Record function id": 0, "Sequence number": 125, "Fwd thread id": 1, "Ev Idx": 884
    }
  },
  {
    "ph": "X", "cat": "cpu_op", "name": "ViewBackward0", "pid": 112835, "tid": 112835,
    "ts": 5284315312690.610, "dur": 17.072,
    "args": {
      "External id": 886,"Record function id": 0, "Sequence number": 125, "Fwd thread id": 1, "Ev Idx": 885
    }
  }
...
</code></pre>
<p>how do I determine to which viewbackward operation in the whole model does the slice with external id 886 belong to?</p>
<p>I am guessing determining the flow of data would help here but I find it challenging to trace the flow of data through the model solely using Chrome tracing tools. Has anyone encountered a similar issue or have suggestions on how to better understand and determine dataflow just based on the chrome trace.</p>
<p>Any insights or recommendations would be greatly appreciated.</p>
Request for Python Tasks in PyTorch Project

Mohammad_Salehi: <p>I hope this message finds you well. I am Mohammad Salehi, and I am eager to contribute to the PyTorch project. I am proficient in Python and have experience working with PyTorch itself, but I am not familiar with the C++ parts of the codebase.</p>
<p>Could you kindly assign me issues or tasks that primarily involve Python, or guide me towards areas where Python-based work is needed? I am keen to help where my skills can make the most impact without delving into C++ related areas.</p>
<p>I am looking forward to contributing and being a part of this amazing project.</p>

ptrblck: <p>Hi Mohammad! It’s great seeing your interest in contributing to PyTorch. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
You could start by looking at GitHub issues tagged with the <a href="https://github.com/pytorch/pytorch/labels/good%20first%20issue"><code>good first issue</code> label</a> and check which ones would be focused on Python. Once you’ve found an interesting one you should add a comment to the issue stating you would try to fix it. The <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">PyTorch Contribution Guide</a> is also a good resource to get started.</p>
Parallel sampling with torch.vmap, confusion on the notion of "batched" distributions

mathis_c99: <p>Hi all,</p>
<p>I’m trying to port some Jax code to Pytorch, so I’m naturally turning to <code>torch.vmap</code> to avoid rewriting a lot of stuff in a different way. However I already have an issue. I’m building hidden Markov models (or state-space models) and one thing I do a lot is to sample from transition kernels, basically functions that apply a deterministic map + some noise to a given input. I have a problem with the randomness part because I’m confused with how torch distributions work. I wrote a <code>Noise</code> class as follows</p>
<pre><code class="lang-auto">import torch.distributions as dists 
class Noise:
    def __init__(self, 
                 noise_dist:dists.Distribution, 
                 noise_fn:Callable):
        self.noise_dist = noise_dist 
        self.noise_fn = noise_fn

    def __call__(self, x, params):
        random_sample = self.noise_dist(**params['dist']).rsample()
        return self.noise_fn(x, random_sample, params['fn'])
</code></pre>
<p>Now suppose I want to apply simple additive gaussian noise to a vector of samples of dimension <code>d</code>, so I wrote:</p>
<pre><code class="lang-auto">import torch 


AdditiveGaussianNoise = Noise(noise_dist = dists.MultivariateNormal, 
noise_fn = lambda x,y,_: x+y)

noise_params = {'dist':{'loc':torch.zeros(d), 
                            'covariance_matrix':torch.eye(d)}, 'fn':{}}
</code></pre>
<p>Suppose I’m now trying to apply that noise to a vector of samples, I do</p>
<pre><code class="lang-auto">clean_samples = torch.ones(num_samples,d)
noisy_samples = torch.vmap(Noise.apply, in_dims=(0,None), randomness='different')(clean_samples, noise_params)
</code></pre>
<p>I get the following error “RuntimeError: vmap: Cannot ask for different inplace randomness on an unbatched tensor. This will appear like same randomness. If this is necessary for your usage, please file an issue with functorch.”</p>
<p>My questions are the following:</p>
<ol>
<li>Why is <code>clean_samples</code> considered “unbatched” ? The first dimension is effectively a batching dimension, and the second is the dimension of the sampling space. Is there are particular reshaping I should be doing ?</li>
<li>How could I make it work ?</li>
</ol>
<p>Thanks in advance!</p>

fl1ps1de5: <p>Hi there - experiencing something similar. Wondering how/if you ever got around this?</p>
Model.eval() and model.train() switching back and forth

L_Z: <p>My model contains dropout layers. So I just want to double check, when I am looping through the number of epochs, I want to get both train and validation’s loss and accuracy. So I have model.train() … and then model.eval() in side the loop.<br>
And then for my testing, I also have model.eval().<br>
Is my implementation correct?</p>

Tony-Y: <p>The answer can be found on PyTorch tutorial:<br>
<a href="https://pytorch.org/tutorials/beginner/introyt/trainingyt.html#per-epoch-activity" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/introyt/trainingyt.html#per-epoch-activity</a></p>

L_Z: <p>Thank you for sharing the link. But I have not seen example of testing process in it? Just wondering if I still need model.eval() for that.</p>

Tony-Y: <p>You need model.eval() upon testing. The reason is the same as one of validation.</p>
How do you normalize depth data?

ggeof: <p>Hi,</p>
<p>I’m currently working on object detection using RGB and depth data. More specifically, I’d like to start by applying contrastive learning, i.e. training two networks to pass from RGB to depth and vice versa.</p>
<p>Normalizing an RGB image is easy, but for depth data, I don’t know what the best method is for normalizing the data.</p>
<ul>
<li>Convert depth map to grayscale and normalize ?</li>
<li>Convert a tensor directly to meters ?</li>
<li>Normalize all values between 0 and 1.0 directly ?</li>
</ul>
<p>Or is there another way ?<br>
Do you know if any paper has been issued on this issue?</p>
<p>Thanks in advance</p>
Model inference speed

Fatih_Haslak: <p>I have a pipeline where I sequentially run multiple segmentation models, but I only have a single GPU, and this process is running quite slowly. In my project, the number of models is even higher. Is there a way to speed up the execution of these models on a single GPU?</p>
<p>" Model 1 prediction<br>
<span class="hashtag-raw">#pred_v01</span> = model_seg_v01.predict(image)<br>
Some processing with pred_v01</p>
<p>Model 2 prediction<br>
<span class="hashtag-raw">#pred_v02</span> = model_seg_v02.predict(image)<br>
Some processing with pred_v02</p>
<p>Model 3 prediction<br>
<span class="hashtag-raw">#pred_v03</span> = model_seg_v03.predict(image)<br>
Some processing with pred_v03"</p>
<p>++ Platform Windows and Pytorch ++</p>
Can gradient accumulation help better utilize GPU?

raghavm1: <p>I’m wondering on ways to improve GPU utilization.</p>
<p>While I’ve tried to max out batch size, to actually utilize GPU like <code>nvidia-smi</code> helps monitor in its rightmost column, does increasing the number of gradient-accumulation-steps help?</p>
<p>I’m noticing low GPU utilization and was wondering on ways to maximize it without needing to increase batch size.</p>

Soumya_Kundu: <p>From what I’ve this is mostly use-case dependent.</p>
<p>Few previous similar questions here: <a href="https://discuss.pytorch.org/t/how-to-ensure-my-gpu-is-utilized-to-the-fullest/69541" class="inline-onebox">How to ensure my GPU is utilized to the fullest?</a></p>
<p>Most of them are sorta batch size related.</p>
<p>For a few generic attempts, a good guide - <a href="https://pytorch.org/tutorials/recipes/recipes/tuning_guide.html" class="inline-onebox" rel="noopener nofollow ugc">Performance Tuning Guide — PyTorch Tutorials 2.4.0+cu121 documentation</a></p>
<p>I have not come across anything better.</p>
Profiler correlation of forward/backward

TomB: <p>I’m trying to correlate forward/backward operations with the autograd profiler. I’m using nsight systems now that nvprof is EOL but don’t think it’s related to that.<br>
According to the current docs for <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.profiler.emit_nvtx" rel="nofollow noopener"><code>torch.autograd.profiler.emit_nvtx</code></a>  you can correlate them based on a <code>seq=&lt;seq_nr&gt;</code> in forward pass operations and a <code>stashed seq=&lt;seq_nr&gt;</code> in the corresponding backward operations. However in the output in 1.3 I don’t find any <code>stashed seq</code> numbers, just <code>seq</code> numbers. It looks like subsequent refactoring of <code>torch/csrc/autograd/profiler.cpp</code> may have altered this as I can find the <code>stashed seq</code> in <a href="https://github.com/pytorch/pytorch/blob/4cc6c7a6eb125534e2e446e3c5bdcabf553bf94d/torch/csrc/autograd/profiler.cpp#L120" rel="nofollow noopener">earlier code</a> but not in current code.<br>
As I understand it from the docs the <code>stashed seq=</code> is useful to distinguish operations in the backwards which correlate to forward operations from operations in the backward that don’t correlate and will just have a <code>seq=</code> in backward.</p>
<p>Don’t have easily runnable code to post (but can put something together if that will help). Running with the code from the <code>emit_nvtx</code> docs of:</p>
<pre><code class="lang-auto">with torch.cuda.profiler.profile():
  with torch.autograd.profiler.emit_nvtx():
    ...
</code></pre>
<p>Then launching with:<br>
<code>nsys profile -t cublas,cuda,cudnn,nvtx -c cudaProfilerApi --kill=none $(which python) my_script.py</code><br>
Using NVIDIA_Nsight_Systems_Linux_2019.5.1.58 (nvtx option only added in recent versions). As noted I get the <code>seq=</code>  nvtx ranges, just not the <code>stashed seq=</code> so seems like NVTX ranges are working.</p>

Thud1835: <p>At first I thought I had the same problem, but turns out that not every operation in the backward pass is annotated with a sequence number. And the ones that are use <code>seq =</code> instead of <code>stashed seq =</code>.</p>
<p>Someone should probably update the documentation to reflect this.</p>
Is ONNX supported by pytorch geometric?

Carter_F: <p>Does pytorch-geometric support to_onnx? I have a pytorch lightning model that contains multiple torch.nn.Modules, one of which being a GNN using pytorch geometric message passing.<br>
I am able to get lightning’s to_onnx method to run, but it breaks apart when I try to do inference (it also adds input_names other than the ones I defined and changes input dimensions in unexpected ways)<br>
I saw an <a href="https://github.com/pyg-team/pytorch_geometric/discussions/4587" rel="noopener nofollow ugc">old github post</a> saying ONNX is not supported by pytorch-geometric, just wondering if that is still true.</p>
NVIDIA driver on your system is too old (found version 9010)

Mehboob_Riaz: <p>when i try to run my code written for pytorch i got the following warning<br>
“The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver by downloading and installing a new version from the URL: <a href="http://www.nvidia.com/Download/index.aspx" class="inline-onebox" rel="noopener nofollow ugc">Official Drivers | NVIDIA</a> Alternatively, go to: <a href="https://pytorch.org" rel="noopener nofollow ugc">https://pytorch.org</a> to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at C:\actions-runner_work\pytorch\pytorch\builder\windows\pytorch\c10\cuda\CUDAFunctions.cpp:108.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0”</p>

ptrblck: <p>You might need to update the NVIDIA driver based on the error message.</p>

Mehboob_Riaz: <p>Which version of drivers I need to install from Nvidia website as there is a variety of drivers available there?</p>

ptrblck: <p>Select your OS, GPU, etc. <a href="https://www.nvidia.com/download/index.aspx">here</a> and download the latest one.</p>

Mehboob_Riaz: <p>Thanks, I’ll check it out!</p>

Alex_Yan: <p>after download driver the latest(latest driver 2021), still   UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 9010).</p>
<p>torch__version__ : 2.4.1+cu124<br>
nvcc --version: Cuda compilation tools, release 12.6, V12.6.68<br>
Build cuda_12.6.r12.6/compiler.34714021_0</p>
<p>how to fix this?</p>

ptrblck: <p>You might have multiple drivers installed so uninstall the older one first before installing a new one.</p>

Alex_Yan: <p>how to uninstall the older driver properly? because after i uninstall driver from device manager and install the latest, its said windows 10 cannot find the EULA.</p>
OpenCLIP: Visual Encoder Isolation, Memory Management

Zaigham_Randhawa: <p>I am working with pretrained models from OpenCLIP</p><aside class="onebox githubrepo" data-onebox-src="https://github.com/mlfoundations/open_clip">
  <header class="source">

      <a href="https://github.com/mlfoundations/open_clip" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <div class="github-row" data-github-private-repo="false">
  <img width="690" height="344" src="https://opengraph.githubassets.com/6b16cb471a5ff8baf79472db0b825460/mlfoundations/open_clip" class="thumbnail">

  <h3><a href="https://github.com/mlfoundations/open_clip" target="_blank" rel="noopener nofollow ugc">GitHub - mlfoundations/open_clip: An open source implementation of CLIP.</a></h3>

    <p><span class="github-repo-description">An open source implementation of CLIP.</span></p>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Once I load a CLIP encoder, does doing this eliminate the rest of the components from the GPU memory?</p>
<p><strong>model, ,  = open_clip.create_model_and_transforms()</strong><br>
<strong>model = model.visual</strong></p>
<p>Should I be doing stuff like:</p>
<pre><code>import gc

full_model, _, _ = open_clip.create_model_and_transforms()

# Isolate the visual component
visual_model = full_model.visual

# Create a new state dict with only the visual component
visual_state_dict = {k: v for k, v in full_model.state_dict().items() if k.startswith('visual.')}

# Create a new model with only the visual component
isolated_model = torch.nn.Module()
isolated_model.visual = visual_model

# Load the visual state dict into the new model
isolated_model.load_state_dict(visual_state_dict, strict=False)

# Delete references to the full model and unused variables
del full_model, visual_model, visual_state_dict

# Force garbage collection
gc.collect()

# Clear CUDA cache if using GPU
if torch.cuda.is_available():
    torch.cuda.empty_cache()
</code></pre>
<p>Any better way???</p>

Zaigham_Randhawa: <p>I did this and it worked. I was able to get my code running, it didn’t run beforehand. Any insights on torch memory management which might be useful?</p>
<pre><code>            full_model, _, _ = open_clip.create_model_and_transforms()
            visual_model = copy.deepcopy(full_model.visual)
            del full_model
</code></pre>
Limiting precision of specific buffers

Daphna: <p>Similar to: <a href="https://discuss.pytorch.org/t/half-precision-ignoring-a-buffer/75577" class="inline-onebox">Half precision: ignoring a buffer</a>,<br>
I have a model that uses a custom module with several specific buffers which I wish to limit to at least single precision, as half precision is not enough for them. Other than those, the entire model can be converted to half.<br>
What is the best approach? Currently, I’m considering overriding the <code>_apply</code> method in my custom module, but this feels a bit risky and inelegant. Any suggestions?</p>

ptrblck: <aside class="quote no-group" data-username="Daphna" data-post="1" data-topic="209041">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/daphna/48/72504_2.png" class="avatar"> Daphna:</div>
<blockquote>
<p>I’m considering overriding the <code>_apply</code> method in my custom module, but this feels a bit risky and inelegant. Any suggestions?</p>
</blockquote>
</aside>
<p>I assume you are calling <code>.to()</code> directly on your model, which is then transforming all registered parameters and buffers to the desired <code>dtype</code>. If so, you could transform or re-initialize the buffers again as <code>float32</code> tensors. Note that explicitly transforming tensors to <code>torch.float16</code> is not the recommended way and you should use the <code>autocast</code> approach which will keep all buffers and parameters in <code>float32</code>.</p>

Daphna: <p>Yes I am using .half() on the entire model, which I assume is the same as using .to().<br>
Is autocast also the recommended approach for inference, or saving a smaller-sized model? because this is where I encounter accuracy loss.<br>
I am transforming the buffers back to float32 during training, but it will not help to prevent losing precision when saving or loading the model for fp16 inference.</p>

ptrblck: <p>Yes, <code>autocast</code> can also be used during inference and won’t cause any accuracy issues assuming the training worked fine since the same parameters and buffers (in <code>float32</code>) will be used. Manually transforming the parameters and buffers to <code>float16</code> for inference could work, but you would need to verify it based on your use case.</p>
Docker Image for PyTorch 2.4.1 CUDA 12.4 with cuDNN8 Missing

vivekjoshy: <p>I don’t see <code>pytorch/pytorch:2.4.1-cuda12.4-cudnn8-devel</code> image in docker hub. I’m trying to run my pytorch lightning setup on 4 RTX 4090s.I was told RTX 4090 only supports 8.9 compute capability…is this different from cuDNN9 and cuDNN8? Can I use <code>pytorch/pytorch:2.4.1-cuda12.4-cudnn9-devel</code> with RTX 4090?</p>

ptrblck: <aside class="quote no-group" data-username="vivekjoshy" data-post="1" data-topic="209589">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vivekjoshy/48/65444_2.png" class="avatar"> vivekjoshy:</div>
<blockquote>
<p>.I was told RTX 4090 only supports 8.9 compute capability…</p>
</blockquote>
</aside>
<p>That’s wrong since compute capability 8.9 is binary compatible with 8.6 and 8.0.</p>
<aside class="quote no-group" data-username="vivekjoshy" data-post="1" data-topic="209589">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vivekjoshy/48/65444_2.png" class="avatar"> vivekjoshy:</div>
<blockquote>
<p>is this different from cuDNN9 and cuDNN8</p>
</blockquote>
</aside>
<p>Yes, the compute capability of a GPU is not directly related to cuDNN versions (the library itself needs to build their kernels for a compatible GPU architecture but cuDNN8 does not correspond to compute capability 8.x).</p>
CUDA 12.6 being forced by pytorch Conda install

Marin_Katic: <p>Hi, I have Windows 11 and I never installed Cuda 12.6. I have a clean Cuda 11.8 and CuDNN install. When running: conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia, it installs Cuda 12.6 libraries instead Cuda 11.8 libraries. This causes code to get error like RuntimeError: Library cublas64_12.dll is not found or cannot be loaded. The Cuda 12 libraries are: cuda-cccl, cuda-cccl_win-64 and cuda-version 12.6.37. This happens when I run any combination of versions for the command listed above, even when I do not specify the version. If someone could tell me what I am doing wrong I would appreciate it.</p>

ptrblck: <p>Are you using <code>conda-forge</code> as a supported channel? If so, could you remove it and use the default <code>main</code> one instead?</p>

Marin_Katic: <p>Thank you, but I tried conda-forge, I tried pip with pip install torch==2.0.0+cu118 torchvision==0.15.0+cu118 torchaudio==2.0.0+cu118 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a><br>
and I still have the same error since all of these commands are pulling 12.6 libraries.</p>
Problems with gradient flow in a deep unfolded network

LDN: <p>Hi everyone,<br>
I am having some issues with gradient flow in a deep unfolded network being an analytical variation of LISTA (ALISTA), where basically in the following recursion<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b69da90e41cfe3ef60e454a5a5459db01b85ce8.png" alt="image" data-base62-sha1="aL8Bg2x9MZd2CcyvRVsW6SS91gI" width="310" height="36"></p>
<p>the only learnable parameters are the threshold in the soft-thresholding operator and the inner stepsize: W is determined by an optimization problem tailored in compressive sensing framework. So my model is implemented as follows:</p>
<pre><code class="lang-auto">class ALISTA(nn.Module):
    def __init__(self, A, K=5):
        super(ALISTA, self).__init__()

        # Number of layers &lt;-&gt; iterations
        self.K = K

        # Parameters
        self.A = A
        self.W = self.W_optimization() 

        self.beta = nn.Parameter(torch.ones(self.K + 1, 1, 1), requires_grad=True)
        self.mu = nn.Parameter(torch.ones(self.K + 1, 1, 1) / torch.linalg.norm(self.A.T @ self.A, 2), requires_grad=True)  

        self.W1 = self.W.T @ self.A
        self.W2 = self.W.T
        
        # Losses when doing inference
        self.losses = []

    def W_optimization(self):

        N, M = self.A.shape
        W = cp.Variable((N, M))

        objective = cp.Minimize(cp.norm(W.T @ self.A.cpu().numpy(), 'fro'))
        constraints = [W[:, m].T @ self.A.numpy()[:, m] == 1 for m in range(M)]
        prob = cp.Problem(objective, constraints)

        prob.solve()

        return torch.from_numpy(W.value).float()

    def _shrink(self, x, beta):
        return beta * F.softshrink(x / beta, lambd=1)

    def forward(self, y, S=None):
        # Initial estimation with shrinkage
        x = self._shrink(self.mu[0, :, :] * y @ self.W2.t(), self.beta[0, :, :])

        for i in range(1, self.K + 1):
            h = x - self.mu[i, :, :] * (x @ self.W1.t() - y @ self.W2.t())
            x = self._shrink(h, self.beta[i, :, :])

            # If ground truth is provided, calculate the loss for monitoring
            if S is not None:
                self.losses.append(F.mse_loss(x.detach(), S.detach(), reduction="sum").item())

        return x
</code></pre>
<p>The problem is that the gradient of the parameters is always 0 and I don’t get why, since the vanilla LISTA I implemented works and is designed very similarly to this. Thanks in advance for your help.</p>
Loading huggingface torchscript from example not working in c++

Carlos_Costa: <p>Hi all</p>
<p>I just followed this example</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/docs/transformers/en/torchscript">
  <header class="source">

      <a href="https://huggingface.co/docs/transformers/en/torchscript" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://huggingface.co/front/thumbnails/docs/transformers.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/docs/transformers/en/torchscript" target="_blank" rel="noopener nofollow ugc">Export to TorchScript</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It works fine in python, both export and import</p>
<p>but when I try to use it in c++ as explained in step 3 here</p>
<p><a href="https://pytorch.org/tutorials/advanced/cpp_export.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/cpp_export.html</a></p>
<p>It doesn’t work, I am using V2.4.1+cpu in c++ and Python</p>
<p>it is quite simple the import in c++, and no idea why I get “error loading the model”</p>
<p>thanks in advance</p>