akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTrorch Documentation mistake

username: m7md_hka
cooked: <p>Hi,</p>
<p>I believe there is a mistake in the PyTorch documentation regarding <code>v2.RandomCrop</code>. According to the documentation, the <code>fill</code> parameter can be used as follows:</p>
<blockquote>
<ul>
<li><strong>fill</strong> (<em>number</em> <em>or</em> <a href="https://docs.python.org/3/library/stdtypes.html#tuple" rel="noopener nofollow ugc"><em>tuple</em></a> <em>or</em> <a href="https://docs.python.org/3/library/stdtypes.html#dict" rel="noopener nofollow ugc"><em>dict</em></a><em>,</em> <em>optional</em>) – Pixel fill value used when the <code>padding_mode</code> is constant. The default is 0. If a tuple of length 3, it is used to fill the R, G, B channels respectively. The fill value can also be a dictionary mapping the data type to the fill value, e.g., <code>fill={tv_tensors.Image: 127, tv_tensors.Mask: 0}</code> where <code>Image</code> will be filled with 127 and <code>Mask</code> will be filled with 0.</li>
</ul>
</blockquote>
<p>However, when I tried to use it as the documentation suggested:</p>
<p><code>transform = v2.RandomCrop(size=(800, 800), pad_if_needed=True, fill=(0, 0, 255))</code></p>
<p>I encountered the following error:</p>
<pre><code class="lang-auto">RuntimeError: The expanded size of the tensor (4) must match the existing size (3) at non-singleton dimension 0. Target sizes: [4, 103, 906]. Tensor sizes: [3, 1, 1]
</code></pre>
<p>But when I set the fill tuple to <code>fill=(0, 0, 255, 150)</code>, where 150 is the alpha value, it worked.</p>

username: ptrblck
cooked: <p>Are you sure you are loading RGB images or do these contain an alpha channel?</p>

username: m7md_hka
cooked: <p>Oh, it was my mistake for not noticing that my image was in 4 channels (RGBA) instead of 3 (RGB). I have changed the <code>ImageReadMode</code> and its work done.<br>
Thank you very much.</p>
PyTorch Autograds

username: dai_jun
cooked: <p>Hi, everyone! First, I want to say thanks for helping; I got a problem here, in my project, I want to use the autograd of torch in a different way. Simplify my problem is that I want to compute the gradient of output vector with respect to a scalar. For example, in ray tracing, if I ray tracing 4096 rays and I have curvature, which will have impact on all 4096 rays, now I want to compute the gradient of all <strong>4096 rays’ optical path length</strong> with respect to <strong>curvature</strong>;  In simplify, the 4096 rays; optical path length is a tensor (4096, ) and curvature is a tensor (1,)  I want to compute the gradients, every element in (4096, ) with respect to (1, ). However, I found it’s seems like torch always accept a scalar function not a vectors?</p>

username: dai_jun
cooked: <p>BTW, I know can use <em>torch.autograd.functional.jacobian()</em> to accept a vector input, but in my case my function is very complex (the whole ray tracing process), so I have no idea about how to solve my problem. <img src="https://discuss.pytorch.org/images/emoji/apple/dizzy_face.png?v=12" title=":dizzy_face:" class="emoji" alt=":dizzy_face:" loading="lazy" width="20" height="20"></p>

username: soulitzer
cooked: <p>.grad and .backward also work with non-scalar functions if you explicitly specify a gradient vector, i.e. grad_output</p>

username: dai_jun
cooked: <p>Thanks!  .backward with explicitly specify a gradient vector could work with scalar, however, it will get the sum of gradients. I want to get the gradients in a vector form, not the sum of gradients, just like</p>
<pre><code class="lang-auto">a = torch.tensor([1.], requires_grad=True)
b = torch.tensor([1., 2., 3.])
c = b * a
c.backward(torch.ones_like(c))
a.grad
</code></pre>
<p>What I expect is a tensor like,</p>
<pre><code class="lang-auto">[gradient_0, gradient_1, gradient_1]
</code></pre>
<p>What  I really got was,</p>
<pre><code class="lang-auto">tensor([6.])
</code></pre>
<p>It seems like the sum of the gradients?<br>
Any further suggesstion? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"> <img src="https://discuss.pytorch.org/images/emoji/apple/pleading_face.png?v=12" title=":pleading_face:" class="emoji" alt=":pleading_face:" loading="lazy" width="20" height="20"></p>
Help with understanding if images are getting normlaized correctly and its effect on the network performance

username: Foram
cooked: <p>Hello, I am working with the BreakHIs dataset ( Breast Histopathology images), and on computing the mean and standard deviation, I get the following values: mean (0.7941, 0.6329, 0.7694), std=(0.1024, 0.1363, 0.0908). However, when using these values for normalization, I noticed that my normalized image’s min and max values are around -8 and +3, respectively. Are these values correct? How does the incorrect image normalization affect the network performance? Can it cause vanishing gradient issues?</p>
<p>The following is the histogram for one of the original images; it can be seen that the histogram is skewed towards the brighter end.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36684eb97e41bcb19c343aaa917908faa37c2dbe.png" alt="image" data-base62-sha1="7LjgcaAmp9R9pWVkkxFVaNwYUoe" width="483" height="376"></p>
Using torch.compile twice on a model on the same machine, is there a cache of optimized operations?

username: LewsTherin
cooked: <p>I’m using torch.compile to compile a torch model by:</p>
<pre><code class="lang-auto">self.model = torch.load(saved_model_path, map_location=self.device).to(self.device)
self.model.eval()

self.model.half()

# Configure hidet to use tensor core and enable tuning
hidet.torch.dynamo_config.use_tensor_core(True)
hidet.torch.dynamo_config.search_space(2)
self.model = torch.compile(self.model, backend="hidet")
</code></pre>
<p>I ran this on a remote machine a first time. Lots of optimization operations were being skipped because they weren’t supported:</p>
<pre><code class="lang-auto">[2024-02-03 12:48:24,408] torch._dynamo.convert_frame: [WARNING]     raise NotImplementedError("\n".join(lines))
[2024-02-03 12:48:24,408] torch._dynamo.convert_frame: [WARNING] torch._dynamo.exc.BackendCompilerFailed: backend='hidet' raised:
[2024-02-03 12:48:24,408] torch._dynamo.convert_frame: [WARNING] NotImplementedError: The following modules/functions are not supported by hidet yet:
[2024-02-03 12:48:24,408] torch._dynamo.convert_frame: [WARNING]   torch.nn.AvgPool3d
</code></pre>
<p>but many others were. After a while, the optimizations were done and inference was performed.</p>
<p>Then, I ran it again on the same machine, but I only saw the warnings regarding the skipped operations. Why is that? Is there some sort of cache, so that optimizations are not performed again for the same model?</p>
Where can I find the source code of "max_pool2d"

username: Yiru_Shen
cooked: <p>Hi there,</p>
<p>I’d like to find the source code of “max_pool2d” and find that actually Pytorch uses  aten::max_pool2d_with_indices() to perform the operation, which is under the directory of aten/src/ATen/native/Pooling.cpp. See the code below:</p>
<pre><code class="lang-auto">std::tuple&lt;Tensor, Tensor&gt; max_pool1d_with_indices(
    const Tensor&amp; self,
    IntArrayRef kernel_size,
    IntArrayRef stride,
    IntArrayRef padding,
    IntArrayRef dilation,
    bool ceil_mode) {
  if (stride.empty()) {
    stride = kernel_size;
  }
  checkDim("max_pool1d", TensorArg(self, "self", 1), 3);
  check1d("max_pool1d", "kernel_size", kernel_size);
  check1d("max_pool1d", "stride", stride);
  check1d("max_pool1d", "padding", padding);
  check1d("max_pool1d", "dilation", dilation);

  Tensor output, indices;
  std::tie(output, indices) = at::max_pool2d_with_indices(
      self.unsqueeze(2),
      {1, kernel_size[0]},
      {1, stride[0]},
      {0, padding[0]},
      {1, dilation[0]},
      ceil_mode);

  return std::make_tuple(output.squeeze(2), indices.squeeze(2));
}
</code></pre>
<p>However, it calls at::max_pool2d_with_indices() and where can I find this function specifically?<br>
I know Pytorch uses Aten library for math computation but it seems that max_pool2d_with_indices() is the custom function using some basic maths.</p>
<p>Thanks!</p>

username: pu239
cooked: <p>Hey, did you ever find out where this was?<br>
I have also been trying to dig through the code, this is the only other question.</p>
PyTorch + CUDA 11.4

username: deutschgraf
cooked: <p>Hi all!</p>
<p>Does Pytorch supports CUDA 11.4 toolkit? Or could I build pytorch with cuda 11.4 support?</p>

username: ptrblck
cooked: <p>Yes, you can build PyTorch from source using all released CUDA versions between 10.2 and 11.5.</p>

username: umfengxx
cooked: <p>For older verion pytorch, such as 1.1.0, is it possible to build from soure with CUDA 11.4?</p>

username: ptrblck
cooked: <p>We are not backporting new library support for old releases. The already released PyTorch versions are supporting the CUDA toolkits which were supported at that time.<br>
With that being said, if no changes e.g. between CUDA 11.3 and 11.4 were needed, you might be able to use the newer CUDA toolkit, but there is no guarantee.</p>

username: koprupu62
cooked: <p>Pytorch for CUDA 11.4 can’t be build because MAGMA-CUDA114 is needed from <a href="https://anaconda.org/pytorch/" class="inline-onebox" rel="noopener nofollow ugc">pytorch :: Anaconda.org</a> but it does not exist. Is there any solution? I’m working in a VM with vGPU 13.1 that supports CUDA 11.4 and I can’t change the drivers because I’m not not admin.</p>

username: ptrblck
cooked: <p>MAGMA is not a requirement and PyTorch can be built with CUDA11.4.<br>
You could try to either install MAGMA with CUDA11.3, 11.5, or build it from source, if needed.</p>

username: Anar
cooked: <p>I successfully accessed GPU on <strong>CUDA 11.4</strong> and <em>NVIDIA GeForce RTX 2080 Ti</em></p>
<p><code>pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html</code></p>

username: ptrblck
cooked: <p>Your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension. PyTorch binaries ship with their own CUDA dependencies and in your case 11.1 will be used.</p>
Where is the sd.reset in CudaIPCSentDataLimbo::collect

username: jiangxiaobin96
cooked: <p>I can not find the function <code>rd.reset()</code>. Where is the following code?</p>
<pre><code class="lang-auto">bool CudaIPCSentDataLimbo::collect() {
  bool freed_memory = false;
  std::vector&lt;std::unique_ptr&lt;CudaIPCSentData&gt;&gt; reset_blocks;
  { // Begin critical section to modify shared blocks
    std::lock_guard&lt;std::mutex&gt; lock(limbo_mutex_);
    std::vector&lt;std::unique_ptr&lt;CudaIPCSentData&gt;&gt; kept_blocks;
    for (auto&amp; sd : shared_blocks_) {
      if (sd-&gt;counter_value() &gt; 0) {
        kept_blocks.push_back(std::move(sd));
      } else {
        freed_memory = true;
        reset_blocks.push_back(std::move(sd));
      }
    }
    shared_blocks_ = std::move(kept_blocks);
  }
  // Need to reset blocks out of the critical section here, otherwise it deadlocks.
  for (auto&amp; sd : reset_blocks) {
    sd.reset();
  }
  return freed_memory;
}
</code></pre>
CUDA out of memory with adding operation

username: kai422
cooked: <p>Hello everyone,</p>
<p>I’ve recently encountered a CUDA out of memory issue in my project when an “adding operation” is performed. I have no idea why “adding” will cause so much memory cost.<br>
As two tensors only cost ~1MB.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/810f8ea40fea1f1a6a286b77e718d8d7f5fd7b89.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/810f8ea40fea1f1a6a286b77e718d8d7f5fd7b89" title="IMG_4551"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/810f8ea40fea1f1a6a286b77e718d8d7f5fd7b89_2_673x499.jpeg" alt="IMG_4551" data-base62-sha1="ipIVkZtlq4bYMBwKjpfo9yHZKoV" width="673" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/810f8ea40fea1f1a6a286b77e718d8d7f5fd7b89_2_673x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/810f8ea40fea1f1a6a286b77e718d8d7f5fd7b89_2_1009x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/1/810f8ea40fea1f1a6a286b77e718d8d7f5fd7b89.jpeg 2x" data-dominant-color="171A20"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_4551</span><span class="informations">1263×937 275 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Do you have any suggestions on how to debug this issue?</p>

username: ptrblck
cooked: <p>Check the shapes of both tensors as they might be broadcasted as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1024**3//4, 1, device="cuda")
y = torch.randn(1, 1024**3//4, device="cuda")
print(torch.cuda.memory_allocated()/1024**3)
# 2.0

z = x + y
# OutOfMemoryError: CUDA out of memory. Tried to allocate 268435456.00 GiB
</code></pre>

username: kai422
cooked: <p>Thank you so much! Indeed the size is not the same. The first tensor is of size (100000,3), the second tensor is of size (100000,1,3). I never thought it’s due to broadcasting.</p>
Decrease loss function

username: Kallel_Mariem
cooked: <p>Hello, my task is multimodal medical image fusion, the model i work on is complex, but i have a very small data, the train_loss vary between 1.7 and 2.3 and the test_loss varry between 3 and 3.7 what is mean these results? and how can i decrease the loss function ? Thank you</p>
Does the order of graph_module.graph.nodes affect performance?

username: ktexb
cooked: <p>When I get a graph module using torch.compile, the node list in graph_module.graph.nodes has to be in topological order. I read somewhere that that’s considered the “execution order.” If I change the order of those nodes while maintaining topological order, does that impact the performance of the graph?<br>
Are there any drawbacks at all to modifying the order of those nodes, given that topological ordering is maintained?</p>
What are the benefits to limiting param_group size?

username: ringohoffman
cooked: <p>As part of <code>deepspeed</code>’s mixture of experts implementation, they separate expert parameters into their own separate param group via the function <a href="https://github.com/microsoft/DeepSpeed/blob/76ec8b4927974431a4e5de476ecacafceda2abe4/deepspeed/moe/utils.py#L71-L74" rel="noopener nofollow ugc"><code>split_params_into_different_moe_groups_for_optimizer</code></a>. It has the parameter <code>max_group_size = 178956971</code>. What would be the reasoning for limiting param group size? Does this help in a distributed setting or something? What are the benefits?</p>

username: tom
cooked: <p>Per the PR that introduced this splitting, <a href="https://github.com/microsoft/DeepSpeed/pull/2079" class="inline-onebox" rel="noopener nofollow ugc">Improving memory utilization of Z2+MoE by siddharth9820 · Pull Request #2079 · microsoft/DeepSpeed · GitHub</a> , it is to save memory because only one of these groups will be on the GPU in full precision at a time.</p>
<p>Please take the answer with a grain of salt, the question just piqued my curiosity.</p>
<p>Best regards</p>
<p>Thomas</p>
RuntimeError: GET was unable

username: 526
cooked: <p>I have RuntimeError when running code using pytorch.<br>
My CUDA version on both nvidia-smi and nvcc --version are 12.0<br>
PyTorch version: 2.3.0.dev20240202+cu121<br>
Torchvision version: 0.18.0.dev20240202+cu121<br>
Previously, I installed cudatoolkit=11.2.2 cudnn=8.1.0.<br>
Here is my code:</p>
<pre><code class="lang-auto">import easyocr
import cv2
import numpy as np

def detect_text(image_path):
    reader = easyocr.Reader(['vi'])
    img = cv2.imread(image_path)
    result = reader.readtext(img)
    return result

result = detect_text('.png')

for res in result:
    text = res[1]
    print(text)
</code></pre>

username: ptrblck
cooked: <p>Could you uninstall the manually installed CUDA toolkit and cuDNN lib and let PyTorch use the one which ships in it’s binary?</p>

username: 526
cooked: <p>After uninstalling: RuntimeError: cuDNN version incompatibility: PyTorch was compiled  against (8, 9, 2) but found runtime version (8, 1, 0). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN. one possibility is that there is a conflicting cuDNN in LD_LIBRARY_PATH.</p>

username: ptrblck
cooked: <p>Did you remove your locally installed cuDNN from <code>LD_LIBRARY_PATH</code> as suggested?</p>

username: 526
cooked: <p>It worked when I removed locally installed cuDNN from <code>LD_LIBRARY_PATH</code>. Thank you!</p>
Memory issues with gradients in torch.func.grad

username: armal
cooked: <p>Hi all,</p>
<p>I am currently having some memory issues and I think it is due to my use of torch.func.grad and torch.func.vmap. The code currently looks like this:</p>
<pre><code class="lang-auto">models = []
with torch.no_grad(): 
    for i in range(len(x)):
        new_params = x[i]
        vector_to_parameters(new_params, model.parameters())
        model_copy = copy.deepcopy(model)
        models.append(model_copy)
        del model_copy 
    params, buffers = stack_module_state(models)
    base_model = copy.deepcopy(models[0])
del models

def fmodel(params, buffers, mini_batches):
    # return functional_call(base_model, (params, buffers), (x, ))
    return functional_call(base_model, params, mini_batches)

def compute_loss(params, buffer, inputs, labels):
    outputs = fmodel(params, buffers, inputs)
    return -crit(outputs, labels)

batch_indices = torch.tensor(batch_indices)
mini_batches = X[batch_indices]
labels = Y[batch_indices]

ll = vmap(compute_loss)(params, buffers, mini_batches, labels)

ll_grads = vmap(grad(compute_loss))(params, buffers, mini_batches, labels)
concat_grads = torch.cat([torch.tensor(value).view(len(x), -1) for value in ll_grads.values()], dim=1)
return concat_grads
</code></pre>
<p>After I calculate the gradients, I just need the value of the gradients. Not to store the computation graph. I have tried wrapping functions in with torch.no_grad() but this causes my gradientsa values to go to 0 when calculating them.</p>
<p>if anyone has any idea how to fix this, it would be much appeciated!</p>
<p>Kind regards</p>

username: armal
cooked: <p>I managed to fix these memory issues by changing the following:</p>
<pre><code class="lang-auto">models = []
with torch.no_grad(): 
    for i in range(len(x)):
        new_params = x[i]
        vector_to_parameters(new_params, model.parameters())
        model_copy = copy.deepcopy(model)
        models.append(model_copy)
        del model_copy 
    params, buffers = stack_module_state(models)
    base_model = copy.deepcopy(models[0])
</code></pre>
<p>We need to use requires_grad_ on the parameters and set it to False. This fixed my memory issues.</p>
<pre><code class="lang-auto">models = []
with torch.no_grad(): 
    for i in range(len(x)):
        new_params = x[i]
        vector_to_parameters(new_params, self.model.parameters())
        model_copy = copy.deepcopy(self.model)
        models.append(model_copy)
        del model_copy 
    params, buffers = stack_module_state(models)
    for param in params.values():
        param.requires_grad_(False)
    base_model = copy.deepcopy(models[0])
</code></pre>
<p>This still needs to be cleaned up a lot</p>
Pytorch models trained on not-so-large datasets

username: Joppe_De_Jonghe
cooked: <p>When looking at the pretrained classification models from pytorch, they all seem to be trained on very large datasets such as ImageNet, etc.</p>
<p>Is it possible for me to find some pretrainend models somewhere that are compatible with pytorch and are trained with datasets that aren’t extremely large (so that I can use them on my laptop)? Preferably classification models. I am planning to use these models to run some compression experiments for which I need part of the training data, as well as a test or validation set.</p>

username: alepistola
cooked: <p>Maybe I didn’t understand the question well, wouldn’t it be more convenient to use Google Colab (you can upload the data to G Drive).<br>
Alternatively, locally, you can use</p>
<ul>
<li>all the models from <a href="https://pytorch.org/vision/stable/models.html" rel="noopener nofollow ugc">torchvision.models</a></li>
<li>the TIMM library</li>
<li><a href="https://github.com/qubvel/segmentation_models.pytorch" rel="noopener nofollow ugc">smp</a> it may be no longer updated</li>
</ul>
MaNet finetuning (binary semantic segmentation task)

username: alepistola
cooked: <p>Hello everyone,</p>
<p><strong>Introduction</strong>: I am currently working on a computer vision problem, I have satellite images and I have to detect a particular archeological structure (Tell).<br>
I have access to the previously made researches and I have access to a Manet model trained on 5 thousand images of mesopotamic area.<br>
My project objective is to fine-tune this model in order to detect the same archeological structure in different geographical areas which have different characteristics, the problem is that I have something like 200 images of the “new” area which is why I am trying to fine-tune the old model I have access to.</p>
<p><strong>The problem in few words</strong> is: which layers do you usually add on top of a base model for finetuning?</p>
<p>All the tutorial I have found online are for classification problems, however, the output of my model is a mask in (N, H, W) → P format where N is the batch size, and H and W are respectively height and width whereas p is the probability that the particular pixel belongs to the “Tell” or not (binary semantic segmentation). So if you have better tutorials I can follow it would be great. I did not find any tutorial about making a custom segmentation head to concatenate over an existing segmentation model like Unet or MaNet.</p>
<p>Thats what I am now thinking to do, is it sound correct to you?</p>
<ol>
<li>freeze old model weights</li>
<li>add GlobalAveragePooling2D layer</li>
<li>add Dropout(0.2) layer</li>
<li>add Dense(512, 512) (imgs size)</li>
<li>fine tuning</li>
<li>unfreeze old model weights</li>
<li>finetune the entire model with very low lr</li>
</ol>
<p>While I know the whole workflow is correct, the thing that worries me the most is the process of editing the model (points 2-3-4), does it make sense?</p>
<p>Thats the model summary with batch size 8 and image channels = 6:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7ab562f7f316b1fef874ec05f763614886d9fd3.png" data-download-href="https://discuss.pytorch.org/uploads/default/b7ab562f7f316b1fef874ec05f763614886d9fd3" title="model"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b7ab562f7f316b1fef874ec05f763614886d9fd3.png" alt="model" data-base62-sha1="qcOsMnok2TgXCWPAGXbXxamae0b" width="690" height="394" data-dominant-color="3F3F3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">model</span><span class="informations">1033×591 16.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Knowledge distillation, what loss

username: AlbertZeyer
cooked: <p>For knowledge distillation (KD), a quick search revealed many different variants on what loss is used, and other variations.</p>
<p>E.g.:</p>
<ul>
<li>Cross entropy (CE), for example here:
<ul>
<li><a href="https://pytorch.org/tutorials/beginner/knowledge_distillation_tutorial.html#id3" class="inline-onebox" rel="noopener nofollow ugc">Knowledge Distillation Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a></li>
<li><a href="https://en.wikipedia.org/wiki/Knowledge_distillation" class="inline-onebox" rel="noopener nofollow ugc">Knowledge distillation - Wikipedia</a></li>
<li>Distilling the Knowledge in a Neural Network, <a href="https://arxiv.org/pdf/1503.02531.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1503.02531.pdf</a></li>
</ul>
</li>
<li>Kullback-Leibler (KL) divergence, for example here:
<ul>
<li><a href="https://github.com/haitongli/knowledge-distillation-pytorch/blob/master/model/net.py#L100" class="inline-onebox" rel="noopener nofollow ugc">knowledge-distillation-pytorch/model/net.py at master · haitongli/knowledge-distillation-pytorch · GitHub</a></li>
</ul>
</li>
<li>Mean squared error (MSE) of the logits
<ul>
<li>see comparison paper below</li>
<li>“Distilling the Knowledge in a Neural Network” mentions that it can be similar</li>
</ul>
</li>
<li>Jensen–Shannon divergence maybe?</li>
</ul>
<p>Comparisons:</p>
<ul>
<li>Comparing Kullback-Leibler Divergence and Mean Squared Error Loss<br>
in Knowledge Distillation, <a href="https://www.ijcai.org/proceedings/2021/0362.pdf" rel="noopener nofollow ugc">https://www.ijcai.org/proceedings/2021/0362.pdf</a></li>
</ul>
<p>On KL vs CE: Yes, I know it’s the same up to an additional constant (if you consider the teacher prob as constant). So, it should not really make a difference. But still, is there any reason why to choose one over the other?</p>
<p>In all cases, the KD loss is used together with the normal loss (e.g. cross entropy to the ground truth targets), with some loss scales/weighting.</p>
<p>Often, I see the additional factor 1/temperature^2 for the KD loss.</p>
<p>Some related discussion:</p>
<ul>
<li><a href="https://www.reddit.com/r/MachineLearning/comments/ke1r2q/r_why_do_knowledge_distillation_papers_always_use/" rel="noopener nofollow ugc">https://www.reddit.com/r/MachineLearning/comments/ke1r2q/r_why_do_knowledge_distillation_papers_always_use/</a></li>
</ul>
Pytorch docker image significant size increase in newer version

username: sonnehansen
cooked: <p>Hi,</p>
<p>I am building a Docker image for training a mask2former and notice a significant increase in base image size:<br>
pytorch/pytorch 2.1.1-cuda12.1-cudnn8-devel               0b705662863d        2 months ago        16.6GB<br>
pytorch/pytorch 2.0.1-cuda11.7-cudnn8-devel               42a0e9b621e2        8 months ago        13.2GB</p>
<p>In the end, adding my dependencies (transformers, lightning and a few others), and keeping everything else equal I get a final image ~15GB for the older base image and ~ 20GB for the latest. I also notice that other <a href="https://discuss.pytorch.org/t/sudden-surge-in-cpu-ram-usage-after-upgrading-pytorch-v1-6-to-v1-9/181074/3">allude to the issue</a></p>
<p>I am unsure how I should approach the topic if I want to understand the reason for the different. Does anyone have insights that can help me on the way?</p>
<p>Of course I would like to be on the latest version but 20GB prohibitively larger for my training deployment</p>
<p>Thanks</p>
Incorrect Class label for Caltech101 dataset

username: Mayank_Pal
cooked: <p>Hi<br>
I am using <a href="https://pytorch.org/vision/main/generated/torchvision.datasets.Caltech101.html" rel="noopener nofollow ugc">Caltech101</a> datasets from torchvision Pytorch library.<br>
I am trying to build a subset of this large dataset with just five classes (Caltech5). As part of this, I am selecting few classes and segregating them based on the class index.</p>
<p>However, I am unable to map class index given by Caltech101 dataloader to the list of class label correctly.</p>
<p><strong>Class labels</strong>: <a href="https://raw.githubusercontent.com/tensorflow/datasets/master/tensorflow_datasets/testing/metadata/caltech101/3.0.0/label.labels.txt" rel="noopener nofollow ugc">caltech101_class_labels.txt</a></p>
<p>The very first image given by the dataloader itself is not matching.</p>
<p><strong>For example</strong>:</p>
<pre><code class="lang-auto">dataloader = datasets.Caltech101(root="./", download=True)
idx = 0
img, _class = dataloader[idx]
print("Class: ", _class)
plt.imshow(img)
plt.show()
</code></pre>
<p><strong>Output</strong>:<br>
Files already downloaded and verified<br>
Class:  0<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d071678cde9a70d36ce5e9f9dfd0d8131c011e0a.jpeg" alt="image" data-base62-sha1="tJYhtt2CJmfZseY3tGxVhmjV6LE" width="555" height="376"></p>
<p>Class index 0 should belong to <em>accordion</em> but it is a face.</p>
<p>Am i missing anything here ?</p>

username: Bridget_Murphy
cooked: <p>Good luck on your project.</p>
Where is PyTorch source code for bilinear interpolation for image resizing? Interested to code it from scratch

username: tlim
cooked: <p>I cannot seem to backtrack from their libraries import to find the source code for the actual code of bilinear interpolation for image resize.</p>
<p>Tracking from <code>torchvision.transforms.resize</code>, I end up at <code>torch.nn.functional</code> where the <code>interpolate</code> function is imported from: (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/functional.py#L3536" class="inline-onebox" rel="noopener nofollow ugc">pytorch/functional.py at master · pytorch/pytorch · GitHub</a>).</p>
<p>From here, I am lost. For bilinear interpolation (<code>line 3722</code>), it calls <code>torch._C._nn.upsample_bilinear2d()</code>. I believe this is the function that is typically called when someone does image resizing (default mode) in PyTorch.</p>
<p><strong>Searching the code for <code>torch._C._nn.upsample_bilinear2d()</code>, I reach a dead-end here: (<a href="https://github.com/pytorch/pytorch/blob/master/torch/_C/_nn.pyi.in" class="inline-onebox" rel="noopener nofollow ugc">pytorch/_nn.pyi.in at master · pytorch/pytorch · GitHub</a>)</strong></p>
<p><em>The main reason for my inquiry is that I want to follow PyTorch bilinear interpolation (resizing image) algorithm for my CUDA code, I am currently writing a CUDA code that does batch multi-crop and  resizing simultaneously which is not possible with PyTorch GPU code as you can only crop images sequentially and resize the crop sequentially too (2 individual serial process). A lot of bilinear interpolation for images seems to differ in their implementation.</em></p>

username: ptrblck
cooked: <p>The CUDA code for <code>upsample_bilinear2d</code> should be located <a href="https://github.com/pytorch/pytorch/blob/8871ff29b743948d1225389d5b7068f37b22750b/aten/src/ATen/native/cuda/UpSampleBilinear2d.cu">here</a>.</p>

username: tlim
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, do you know where is the CPU version of the code?</p>

username: ptrblck
cooked: <p>I think the CPU implementation is defined <a href="https://github.com/pytorch/pytorch/blob/8871ff29b743948d1225389d5b7068f37b22750b/aten/src/ATen/native/cpu/UpSampleKernel.cpp">here</a>.</p>

username: tlim
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you for the help. I still cannot seem to get the gist of how the implementation for bilinear interpolation is done for PyTorch. Regardless, thank you for the help!</p>

username: Akanksha_Baghel
cooked: <p>Hey <a class="mention" href="/u/tlim">@tlim</a> , did you find the CPU implementation of Bilinear interpolation.<br>
if yes, could you help as i also need to work on Bilinear case.</p>
How can I make a smaller version of libtorch for deployment?

username: michaeleisel
cooked: <p>I want to deploy a standalone executable, for a variety of Linux execution environments, and use CUDA-enabled libtorch for it. To facilitate this, I want to package everything together in as small a binary as possible. One issue, aside from CUDA libraries, is libtorch_cuda.so. When I use <code>nvprune</code> to try to make it smaller by removing some older architectures, it complains that the .so file is not relocatable, so it fails. Are there any easy ways to get rid of certain archs for libtorch_cuda.so? Strip kernels that I don’t use in practice? Statically link to my own pruned versions CUDA libraries like libcublas?</p>
<p>Any thoughts on this, or other tricks or ideas for reducing size?</p>

username: ptrblck
cooked: <p>You could build PyTorch from source for the desired architectures only and wouldn’t need to prune them. This should also allow you to statically link other dependencies into PyTorch, but I’m unsure how well tested this approach is.</p>
Gradient and tensor initialisation with nn.parameter

username: Pierre_Merriaux
cooked: <p>Hi<br>
I have something I didn’t catch, about gradients “propagation”. If I run the example below it works, I got some grad:</p>
<p>import torch</p>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:<br>
pose_opt_rotation_yaw = torch.nn.Parameter(torch.ones(10,3))<br>
image_idx = 1<br>
yaw = pose_opt_rotation_yaw[image_idx,0:1]<br>
euler_angles = torch.stack((torch.zeros(1),torch.zeros(1),yaw))<br>
euler_angles_ref = torch.tensor([0.0,0.0,0.0])<br>
loss1 = (euler_angles - euler_angles_ref).sum()<br>
loss1.backward()<br>
print(pose_opt_rotation_yaw.grad)</p>
<p>but If replace:<br>
euler_angles = torch.stack((torch.zeros(1),torch.zeros(1),yaw))<br>
by<br>
euler_angles = torch.tensor((torch.zeros(1),torch.zeros(1),yaw), requires_grad=True)</p>
<p>there is no more grad. What is the explanation ? what is the proper way to use tensor init with nn.parameter and constants ?<br>
thanks</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Pierre_Merriaux" data-post="1" data-topic="196455">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pierre_merriaux/48/62202_2.png" class="avatar"> Pierre_Merriaux:</div>
<blockquote>
<p>there is no more grad. What is the explanation ?</p>
</blockquote>
</aside>
<p>Re-creating a new tensor detaches it from the computation graph since you are creating a new leaf variable.</p>
<aside class="quote no-group" data-username="Pierre_Merriaux" data-post="1" data-topic="196455">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pierre_merriaux/48/62202_2.png" class="avatar"> Pierre_Merriaux:</div>
<blockquote>
<p>what is the proper way to use tensor init with nn.parameter and constants ?</p>
</blockquote>
</aside>
<p>You should not re-initialize parameters and instead initialize them once in your model’s <code>__init__</code> method. Using <code>torch.stack/cat</code> is the right approach to combine tensors and parameters in the forward pass.</p>

username: Pierre_Merriaux
cooked: <p>thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
I have checked pose_opt_rotation_yaw is still a leaf in the both case, yaw never, and the difference is on euler_angles.</p>
<p>I will avoid to create tensor in forward.<br>
thanks</p>
Questions about tensor and saving it in Memory

username: m7md_hka
cooked: <p>Hello, I’m trying to understand how tensors are stored in memory. Suppose we have a tensor:</p>
<p>tensor = torch.tensor([1, 2, 3])</p>
<p>The metadata will store information such as the size, shape, and strides, while the storage object will store the data (1, 2, 3).</p>
<p>My questions are:</p>
<ol>
<li>Are the values in the tensor stored at unique offset addresses in memory?</li>
<li>Where is the base address of the storage object mentioned above saved? Is it stored in the metadata? I want to comprehend how the tensor accesses the storage object.</li>
</ol>

username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/how-are-pytorch-tensors-stored-internally/195921/2">This post</a> might be helpful to learn more about PyTorch internals.<br>
The data is stored at the starting address defined in <code>tensor.data_ptr()</code> and each value can be accessed by indexing operations using the size and stride.</p>
Egality comparison - simple tensor

username: Gillian
cooked: <p>Can someone explaine what’s going on here ?<br>
Is it normal that tensor([71211144.]) is equal to both 71211143 and 71211144 ?</p>
<p>t = torch.ones((1))<br>
t<br>
Out[22]: tensor([1.])<br>
t == 1<br>
Out[23]: tensor([True])<br>
t == 2<br>
Out[24]: tensor([False])<br>
t == 0<br>
Out[25]: tensor([False])<br>
t = torch.ones((1))<br>
t = t * 71211144<br>
t<br>
Out[28]: tensor([71211144.])<br>
t == 71211144<br>
Out[29]: tensor([True])<br>
t == 71211143<br>
Out[30]: tensor([True])</p>

username: ptrblck
cooked: <p>Yes, this is expected because of rounding as explained in <a href="https://discuss.pytorch.org/t/unexpected-behavior-of-register-hook-when-increasing-the-batch-size/147221/2">this post</a>.<br>
Also, <a href="https://en.wikipedia.org/wiki/Single-precision_floating-point_format">this Wikipedia article</a> gives you the ranges.</p>
How can I save a convert_pt2e model?

username: poutyface
cooked: <pre><code class="lang-auto"># torch.__version__
# '2.2.0+cu121'

import torch.ao.quantization.quantizer.x86_inductor_quantizer as xiq
from torch.ao.quantization.quantizer.x86_inductor_quantizer import X86InductorQuantizer

m = torch.nn.Linear(5, 5)
example_args = (torch.randn(1, 5),)
pre_aten_dialect = torch._export.capture_pre_autograd_graph(m.eval(), example_args)
quantizer = X86InductorQuantizer()
quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())
pt2e_dialect = prepare_pt2e(pre_aten_dialect, quantizer)
pt2e_dialect(example_args[0])
pt2e_dialect = convert_pt2e(pt2e_dialect)
try:
    torch.save(pt2e_dialect, "model.pt")
except :
    print("error")
model = torch.compile(pt2e_dialect)
torch.save(model, "model.pt")

</code></pre>
<p>I got</p>
<pre><code class="lang-auto"> ...RecursionError: maximum recursion depth exceeded while calling a Python object
</code></pre>
<p>Static quantize:<br>
Since it cannot be saved, it must be calibrated every time.</p>

username: jerryzh168
cooked: <p>yeah it’s here: <a href="https://pytorch.org/tutorials/prototype/pt2e_quant_ptq.html#save-and-load-quantized-model" class="inline-onebox" rel="noopener nofollow ugc">(prototype) PyTorch 2 Export Post Training Quantization — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
Convnext weights

username: David_Valensi
cooked: <p>Hi there,</p>
<p>I’m training a Faster RCNN model with a Convnext backbone. That’s my model creation code:</p>
<pre><code class="lang-auto">backbone = torchvision.models.convnext_large(weights="DEFAULT").features
backbone.out_channels = 1536
anchor_generator = AnchorGenerator(
    sizes=((8, 16, 24, 32),),
    aspect_ratios=((0.5, 0.25, 1/6),)
)
roi_pooler = torchvision.ops.MultiScaleRoIAlign(
    featmap_names=['0'],
    output_size=7,
    sampling_ratio=2
)
model = FasterRCNN(
    backbone,
    num_classes=num_classes,
    rpn_anchor_generator=anchor_generator,
    box_roi_pool=roi_pooler
)
</code></pre>
<p>I followed this tutorial:<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a></p>
<p>My problem is that during training it seems like the model begins the training with randomly initialized weights rather than pretrained weights. The mAP@50:95 begins from nearly zero. In contrast, if I train with a Resnet50 backbone with pretrained weights the mAP@50:95 increases much faster. I also tried training with Resnet50 with random weights and the increase in mAP@50:95 reminds me the training done with Convnext weights.<br>
I should mention that I also tried inputting torchvision.models.convnext_large with pretrained=True, and it didn’t make a difference.</p>
<p>Does anyone have a guess about the origin of the problem? Training with Convnext in theory should reach higher mAP@50:95 but I get the opposite results.</p>
<p>David.</p>

username: Gillian
cooked: <p>Hello David,<br>
Could you try specifiying the weights this way:</p>
<pre><code>convnext_weights = torchvision.models.ConvNeXt_Large_Weights.IMAGENET1K_V1
backbone = torchvision.models.convnext_large(weights=convnext_weights).features
</code></pre>
<p>For me, having a look at weights as image helped me sometimes to be sure to have them loaded:</p>
<pre><code>    weights = model.features[0][0].weight.cpu().data.numpy()
    fig, axes = plt.subplots(8, weights.shape[0]//8)
    for cindex, ax in enumerate(axes.flatten()):
        image = (np.transpose(weights[cindex, ...], (1, 2, 0)) * SCALING + 125)
        image[image&lt;0] = 0
        image[image&gt;255] = 255
        ax.imshow(image.astype(np.uint8))
        ax.set_axis_off()
    plt.savefig(os.path.join(workding_directory, 'filter1_weight'))
    plt.close()
</code></pre>
The NVIDIA driver on your system is too old (found version 9010). Please update your GPU driver by downloading and installing a new version

username: Mehboob_Riaz
cooked: <p>I am trying to use GPU graphic card by nvidia with pytorch but i face this error while attempt to run the code. Plz help to resolve the issue</p>

username: ptrblck
cooked: <p>As the error explains your NVIDIA driver is too old. Update it and it should work.</p>
Wrong Tensor type when using Flash Attention 1.0.9

username: Elias_Urios
cooked: <p>Hi everyone,</p>
<p>I’m triying to make a project for my lectures and I’m trying to compare the speed of FlashAttention 1.0.9 against the attention layers we implemented in class. Since the lectures are oriented for ASR, we’ve built a Transformer which recognizes the digits of an audio and later sums the pronounced digits. However, when I first tried using FA on my project, an error regarding an assert of the datatype of the tensor appeared (since FA uses float16 and I was using float32). I decided then to set the default type to <code>torch.flooat.16</code>. Nevertheless, the mel log spectogram class which I’m using from complains about this type and I think I’ve encountered a wall.</p>
<p>Are there any other ways of using FA 1.0.9 without impacting the network performance (I suspect that casting each batch into float32 and the casting again to float16 isn’t the most optimal approach)</p>
Convert to onnx not match

username: yvanwy
cooked: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/05130a2fb259f0ec32705dd2f4df0a9600f48b85.png" alt="1" data-base62-sha1="ITb9AJM1GqCcRMc62UHcYXvkmF" width="629" height="280"></p>
<p>My model is a unet-like model, the mismatched elements is normal?<br>
besides model structure, what else can cause mismatch?</p>

username: ptrblck
cooked: <p>The relative error seems to be a bit high and you might want to double check both values at this index. Your outputs have a large range and magnitude so the absolute error looks alright.</p>
Computation time of bidirectional RNNs

username: Introvert_Loner
cooked: <p>Hi everyone,<br>
According to Andrew Ng’s Bidirectional RNN:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d83ac6dd0b4c09f979c8bb79bcd482dc8384f20.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/0d83ac6dd0b4c09f979c8bb79bcd482dc8384f20" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d83ac6dd0b4c09f979c8bb79bcd482dc8384f20_2_690x304.jpeg" alt="image" data-base62-sha1="1VyjCegM5D8a0DiCIivfezi2KrK" width="690" height="304" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/d/0d83ac6dd0b4c09f979c8bb79bcd482dc8384f20_2_690x304.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d83ac6dd0b4c09f979c8bb79bcd482dc8384f20.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/d/0d83ac6dd0b4c09f979c8bb79bcd482dc8384f20.jpeg 2x" data-dominant-color="F5F6F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1032×455 86.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The computation of backward directional of Bidirectional RNNs are calculated simultaneously with the computation of the forward direction - (Time complexity of O(n) for the sequence length) or the backward directional is only computed after  the computation of the forward direction - (Time complexity of doubled O(n) for sequence length * num_direction)?</p>
<p>Thank you in advance!</p>
Custom LayerNorm vs PyTorch implementation

username: james5
cooked: <p>Hello,<br>
I stumbled upon the implementation of LayerNorm which was based on <a href="https://github.com/facebookresearch/ConvNeXt/blob/main/models/convnext.py#L138-L143" rel="noopener nofollow ugc">ConvNeXt/models/convnext.py at main · facebookresearch/ConvNeXt · GitHub</a>. I wanted to compare it to GroupNorm and the results are pretty weird. Here is snippet:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class LayerNormFunction(torch.autograd.Function):

    @staticmethod
    def forward(ctx, x, weight, bias, eps):
        ctx.eps = eps
        N, C, H, W = x.size()
        var, mu = torch.var_mean(x, dim=1, keepdim=True, unbiased=False)
        y = (x - mu) / torch.sqrt(var + eps)
        ctx.save_for_backward(y, var, weight)
        y = weight.view(1, C, 1, 1) * y + bias.view(1, C, 1, 1)
        return y

class LayerNorm2d(nn.Module):

    def __init__(self, channels, eps=1e-5):
        super(LayerNorm2d, self).__init__()
        self.register_parameter('weight', nn.Parameter(torch.ones(channels)))
        self.register_parameter('bias', nn.Parameter(torch.zeros(channels)))
        self.eps = eps

    def forward(self, x):
        return LayerNormFunction.apply(x, self.weight, self.bias, self.eps)

torch.manual_seed(0)
x = torch.randn(1, 3, 5, 5)
g = nn.GroupNorm(1, 3)
l = nn.LayerNorm((3, 5, 5))
lc = LayerNorm2d(3)

y_g = g(x)
y_l = l(x)
y_lc = lc(x)
print(f"(y_g - y_lc).pow(2).sum().sqrt() = {(y_g - y_lc).pow(2).sum().sqrt()}")
print(f"(y_g - y_l).pow(2).sum().sqrt() = {(y_g - y_l).pow(2).sum().sqrt()}")
print(f"(y_l - y_lc).pow(2).sum().sqrt() = {(y_l - y_lc).pow(2).sum().sqrt()}")
</code></pre>
<p>Can someone explain why results of <code>LayerNorm2d</code> is different from PyTorchs’ norms?</p>