akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Question about activation checkpoint with FSDP

username: 111146
cooked: <p>I found that PyTorch’s FSDP has its own wrapping function (apply_activation_checkpointing_wrapper) for the activation checkpoint.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/workshops/blob/master/FSDP_Workshop/activation_checkpointing/ac_handler.py">
  <header class="source">

      <a href="https://github.com/pytorch/workshops/blob/master/FSDP_Workshop/activation_checkpointing/ac_handler.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/workshops/blob/master/FSDP_Workshop/activation_checkpointing/ac_handler.py" target="_blank" rel="noopener nofollow ugc">pytorch/workshops/blob/master/FSDP_Workshop/activation_checkpointing/ac_handler.py</a></h4>


      <pre><code class="lang-py">import torch
import os
import torch.distributed as dist
from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import (
    checkpoint_wrapper,
    CheckpointImpl,
    apply_activation_checkpointing_wrapper,
)

from transformers.models.t5.modeling_t5 import T5Block

from functools import partial

non_reentrant_wrapper = partial(
    checkpoint_wrapper,
    offload_to_cpu=False,
    checkpoint_impl=CheckpointImpl.NO_REENTRANT,
)

check_fn = lambda submodule: isinstance(submodule, T5Block)
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/workshops/blob/master/FSDP_Workshop/activation_checkpointing/ac_handler.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I want to know the difference between apply_activation_checkpointing_wrapper and gradient_checkpointing_enable.<br>
When I want to apply activation checkpointing with PyTorch’s FSDP, should I apply the function instead of gradient_checkpointing_enable provided by Huggingface models such as GPT2?</p>

username: agu
cooked: <p>I think that <code>gradient_checkpointing_enable()</code> is HuggingFace’s own built-in method that works because HuggingFace models have manual activation checkpointing calls in the model source code that can be enabled/disabled.</p>
<p><code>apply_activation_checkpointing_wrapper()</code> can work for general models (not just HuggingFace) since the user must pass the criteria for checkpointing. If you are using a HuggingFace model, you can try using the HuggingFace <code>gradient_checkpointing_enable()</code> since those checkpoints have been hand-picked. Though, I am not familiar with the compatibility with FSDP.</p>
Effect of padded sequences in NLP Transformer

username: _Bustion
cooked: <p>I am writing my own implementation of the Transformer according to the paper Attention is All You Need. I have never worked with sequence data, and I am getting confused on the padded inputs. Does the padded values have an effect on the output after getting zeroed by the Embedding layer, when bias is applied to them? If yes, should I apply padding mask after each computation to mask the padded values?</p>
<p>In natural language sentences have variable lengths, and training in machine learning requires batching into matrices where variable length is not allowed. The logical decision would be to pad the shorter sentences with 0’s at the end, feed them into the Embedding layer that has pad_idx = 0, and use the output for further processing. <strong>However</strong>, the linear layers (Feed Forward) in the Transformer contain bias which can add weight to those 0’s. My intuition tells me that it does not affect the weights on back-propagation since the derivative of a constant is zero, but <em>I am wondering if this might have a negative impact on the output of the network</em>.</p>
Map tensor of shape [b,x,y] to [b,x+y]

username: Adir_R
cooked: <p>Hi,</p>
<p>Is there a way in PyTorch to input a network (let’s assume a basic MLP network) a 3-dimensional tensor of shape [batch_size, x, y] and get at the output a 2-dimensional tensor of shape [batch_size, x+y]? I know I can reshape the input to [batch_size, x*y] and then the solution will be simple, but I assume x and y to be high dimensional and I want to avoid this multiplication.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Maybe it won’t fit your use case but you could use an <code>nn.Conv1d</code> layer accepting <code>x</code> input channels with a kernel size of <code>y</code> and returning <code>x+y</code> output channels. You would need to <code>squeeze</code> the spatial dimensions with a size of 1 but the shapes should then fit your requirements.</p>
Torch.where is breaking model training

username: Kinbote
cooked: <p>I’m trying to divide one tensor by another in torch, but only when the values of the denominator exceed a certain threshold.</p>
<p>This implementation works, but won’t compile in torchdynamo.</p>
<pre><code class="lang-auto">wsq_ola = wsq_ola.to(wav).expand_as(wav).clone()
min_mask = wsq_ola.abs() &lt; eps
wav[~min_mask] = wav[~min_mask] / wsq_ola[~min_mask]
</code></pre>
<p>I tried to implement the same thing with <code>torch.where</code> instead as follows:</p>
<pre><code class="lang-auto">wsq_ola = wsq_ola.to(wav).expand_as(wav).clone()
min_mask = wsq_ola.abs() &lt; eps
wav = torch.where(min_mask, wav, wav / wsq_ola)
</code></pre>
<p>Unfortunately, once I make this change, the model no longer converges. Is there some issue with my use of torch.where here? For context, this is part of an stft layer with no trainable weights.</p>

username: ptrblck
cooked: <p>I cannot see any issues using your second code and get the same gradients:</p>
<pre><code class="lang-python">wsq_ola = torch.randn(10, 10, requires_grad=False)
wav = torch.randn(10, 10, requires_grad=True)
eps = 0.1
min_mask = wsq_ola.abs() &lt; eps
out = torch.where(min_mask, wav, wav / wsq_ola)
out.mean().backward()
print(wav.grad.abs().sum())
# tensor(1.9265)


def fun(wav, wsq_ola, eps):    
    out = torch.where(min_mask, wav, wav / wsq_ola)
    return out

wav.grad = None
fun = torch.compile(fun)
out = fun(wav, wsq_ola, eps)
out.mean().backward()
print(wav.grad.abs().sum())
# tensor(1.9265)
</code></pre>
<p>Do you see a difference using my code?</p>

username: Kinbote
cooked: <p>I don’t see any difference when I run your code or in my unit tests. However, when I start an actual training run, suddenly nothing works. Is there an alternate way to express this division? I only need it to get torchdynamo working.</p>
Replicating pytorch reported accuracy for qat mobilenet v3

username: Anirudh_Alameluvari
cooked: <p>I’m trying to replicate the accuracy pytorch reported for the quantization aware trained mobilenet_v3 model. I am running it with fbgemm backend, 128 batch size and ran it for 80 epochs. My best model got an accuracy of 71.596%. Pytorch reports an accuracy of 73.004% on the same. Has anyone been able to reach this accuracy? If so, can you please share your configurations or help me figure out why I’m stuck at this accuracy? I’m running my code on a single  Tesla V100 GPU. It takes ~45 mins for one epoch to complete.</p>
<p>Thanks much!</p>

username: Vasiliy_Kuznetsov
cooked: <p>Here is the recipe: <a href="https://github.com/pytorch/vision/tree/main/references/classification#qat-mobilenetv3" class="inline-onebox" rel="noopener nofollow ugc">vision/references/classification at main · pytorch/vision · GitHub</a></p>
Register submodules

username: Nick95
cooked: <p>Dear all,<br>
I have implemented a CNN composed by several layers using libtorch. I have first defined a generic layer as:</p>
<pre><code class="lang-auto">struct ConvBlock : torch::nn::Module {
    torch::nn::Conv1d conv{nullptr};
    torch::nn::ReLU   relu{nullptr};

    ConvBlock(int in_channels, int out_channels, int kernel_size, int padding):
        conv(torch::nn::Conv1d( torch::nn::Conv1dOptions(in_channels, out_channels, kernel_size).stride(1).padding(padding) ) ),
        relu(torch::nn::ReLU()){
            // Register the two modules of the convolutional block
            conv = register_module("conv", conv);
            relu = register_module("relu", relu);
    }

    torch::Tensor forward(torch::Tensor x){
        x = conv-&gt;forward(x);
        return relu-&gt;forward(x);
    }
};
</code></pre>
<p>Then I have defined the Network model as:</p>
<pre><code class="lang-auto">class CNNNet : public torch::nn::Module {
    public:
        CNNNet(){
            c1 = register_module("c1", c1);
            c2 = register_module("c2", c2);
            c3 = register_module("c3", c3);
            p1 = register_module("p1", p1);
            c4 = register_module("c4", c4);
            c5 = register_module("c5", c5);
            c6 = register_module("c6", c6);
        }

        std::tuple&lt;torch::Tensor, torch::Tensor&gt; forward(torch::Tensor x){
            torch::Tensor indices;
            x = c1.forward(x);
            x = c2.forward(x);
            x = c3.forward(x);
            x = p1-&gt;forward(x);
            x = c4.forward(x);
            x = c5.forward(x);
            x = c6.forward(x);
            return x; 
        }
    private:
        ConvBlock c1 = ConvBlock(1, 16, 7, 0);
        ConvBlock c2 = ConvBlock(16, 16, 5, 0);
        ConvBlock c3 = ConvBlock(16, 16, 3, 0);
        torch::nn::MaxPool1d p1 = torch::nn::MaxPool1d(torch::nn::MaxPool1dOptions(2));
        ConvBlock c4 = ConvBlock(16, 32, 3, 0);
        ConvBlock c5 = ConvBlock(32, 32, 3, 0);
        ConvBlock c6 = ConvBlock(32, 32, 3, 0);
};
</code></pre>
<p>However, when I compile I get the following error:</p>
<blockquote>
<p>error: no matching function for call to ‘ CNNNet::register_module(const char [3], ConvBlock&amp;)’<br>
59 |             c2 = register_module(“c2”, c2);</p>
</blockquote>
<p>How can I manage this issue?</p>
MultiHeadAttention Weights Interpretation

username: alexmehta
cooked: <p>MultiHeadAttention <strong>attn_output_weights</strong> returns (L,S) shape sequence. Where L is the target and S is the source.</p>
<p>What do target and source weights actually mean? I want to get the transformer’s weightage of the input values. How would I do this?</p>

username: _Bustion
cooked: <p>Assuming that you have <code>average_attn_weights=True</code>, the <strong>attn_output_weights</strong> are the transformer’s weightage of the input values (attention matrix used to scale the input values) averaged across different heads as far as I know.</p>
<p>According to Pytorch <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html#torch.nn.MultiheadAttention" rel="noopener nofollow ugc">docs</a>, the L is anything you want to tell the network  to pay attention to, while the S is what you use as an input.</p>

username: alexmehta
cooked: <p>Sorry I’m still a bit confused. In self-attention, what would the matrix look like?</p>

username: _Bustion
cooked: <p>Usually the self-attention matrix looks like a square matrix. The size of it depends on the input since it scales the input according to attention scores.</p>
Access the attributes of inherited PyTorch class in C++ frontend

username: ChaeunLee
cooked: <p>Hi,</p>
<p>I am trying to customize the tensor in PyTorch (inherited from torch.Tensor class) referring to <a href="https://discuss.pytorch.org/t/subclassing-torch-tensor/23754/5">this discussion</a>.</p>
<p>For some reasons, I wish to access the attributes from C++ frontend, for instance, <code>MyObject.extra_data</code> in the above reference. One thing that is interesting is that the customized tensor can be loaded as an argument of C++ frontend with <code>torch::Tensor</code>. e.g.,</p>
<pre><code class="lang-auto">torch::Tensor load_custom_tensor(torch::Tensor my_obj){
    float* my_obj_ptr = my_obj.data_ptr&lt;float&gt;();
    my_obj_ptr[0] += 1;
    return my_obj;
}
</code></pre>
<p>In the above code, <code>my_obj_ptr</code> points the tensor data, and surely the tensor operation works. When the function is bound with python, the return type in the python code is not <code>torch.Tensor</code>, but <code>MyObject</code>, which means that the customized class, <code>MyObject</code>, is well recognized in C++ frontend. But, I cannot find the way to access python attributes in the customized class as members in C++ frontend object.</p>
<p>How could I access these attributes defined in the python code? If impossible, should I override <code>torch::Tensor</code> class in C++ frontend?</p>
Contrastive loss decreases drastically

username: Jeff1
cooked: <p>Currently doing contrastive learning on a dual-stream model with one XLM-RoBERTa and a CLIP-text model, loading the pretrained parameters and adding a new pooler for projecting [CLS], calculating with infoNCE loss.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/1064fa31ed5fc9c9d01535809081f5921bec439f.png" alt="image" data-base62-sha1="2l1YUwzYQFGKmVTFlcYzDdTE0s7" width="211" height="174"></p>
<p>But as the loss curve shows, contrastive loss decreases drastically, reaching near 0 at about 2000 steps (sometimes even faster with larger lr), and if it keeps training, the loss will suddenly jump to about 4, I think it is caused by gradient explosion.</p>
<p>I’ve added autograd and scaler for fp16 training and I think it’s used correctly, here is a snippet.</p>
<pre><code class="lang-auto">def ttc_iter(model, batch, optimizer, scaler, metric_logger, device):
    train_batch = [t.to(device) if t is not None else None for t in batch]

    optimizer.zero_grad()

    with autocast():
        loss = model(train_batch)

    scaler.scale(loss['loss_itc']).backward()
    metric_logger.update(loss_mln_eng_ttc=loss['loss_itc'].item())

    scaler.unscale_(optimizer)
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    scaler.step(optimizer)
    scaler.update()

    return loss
</code></pre>
<p>I couldn’t figure out where the problem occurs, maybe somewhere in the model (since I have added additional encoder layers to the encoder structure)? Thanks for any suggestions in advance!</p>
Difference in attention maps as loss function

username: CindyXWu
cooked: <p>I am having difficulty extracting outputs of intermediate layers for a student and (fixed) teacher model in distillation, and then taking the MSE of these as the training loss. For a start, PyTorch makes it very difficult to extract intermediate layer outputs at all. You either have to edit the forward function of the model or use a forward hook. I found that the latter method (hook) does not work because the output of the layer is not a leaf tensor and cannot require grad. I resorted to using the package torch_intermediate_layer_getter which seems to return an OrderedDict of tensors of layers you specify. Importantly, these come with grad:</p>
<pre><code class="lang-auto">def feature_extractor(model, inputs, return_layers, grad=True):
    """Extract feature maps from model.
    Args:
        model: torch model, model to extract feature maps from
        inputs: torch tensor, input to model
        return_layers: dictionary of layer names to return
    """
    assert inputs.requires_grad
    mid_getter = MidGet(model, return_layers, True)
    mid_outputs, model_outputs = mid_getter(inputs)
    features =  list(mid_outputs.items())[0][1]
    if grad == False:
        features = features.detach()
        return features
    assert features.requires_grad
    return features
</code></pre>
<p>When computing the rest of my loss function, I always check that variables require grad:</p>
<pre><code class="lang-auto">def feature_map_diff(s_map, t_map, aggregate_chan):
    """Compute the difference between the feature maps of the student and teacher models.
    Args:
        s_map: torch tensor, activation map of teacher model [batch_size, num_channels]
        t_map: torch tensor, output of teacher model [batch_size, num_channels]
        aggregate_chan: bool, whether to aggregate the channels of the feature activation
    """
    # Aggregate the channels of the feature activation using root squared absolute value of channels to create activation map
    if aggregate_chan:
        s_map = torch.sqrt(torch.sum(torch.abs(s_map)**2, dim=1))
        t_map = torch.sqrt(torch.sum(torch.abs(t_map)**2, dim=1))
    assert s_map.requires_grad
    # Compute the difference between the feature maps
    # loss = F.mse_loss(s_map, t_map, reduction='mean').requires_grad_()
    loss = torch.mean((s_map/torch.norm(s_map, p=2, dim=-1).unsqueeze(-1) - t_map/torch.norm(t_map, p=2, dim=-1).unsqueeze(-1))**2 )
    assert loss.requires_grad
    return loss
</code></pre>
<p>None of these assertions raise any error, but in my main training loop, with the following after <code>loss.backward()</code> and <code>optimizer.step()</code>:</p>
<pre><code class="lang-auto">for param in student.parameters():
    assert param.requires_grad
</code></pre>
<p>I end up with an error. Can someone explain to me what is going on and how I can achieve what I’m trying to do better?</p>
High gap between losses

username: goksu
cooked: <p>Hello;</p>
<p>I have an unsupervised network model to train with unlabeled datasets. While the model is training my learning curves are like the figure below. I use a similar model like ResNet18 with a smaller kernel size. I changed many hyperparameters like learning rate, weight_decay, batch size etc. Also, adding many augmentations to get more complex data is not working. Do you have any opinion to solve this problem?</p>
<p>Thank you!!!</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/4/a4de07b790682a1b4d4dc3171a92d696665c23a1.png" alt="train-val-fig-task4" data-base62-sha1="nwu09MuTF9QQXveGoHgz87WMuCR" width="386" height="262"></p>
<p>Cheers,<br>
G.</p>
Different CUDA versions bring different results

username: Shino
cooked: <p>I have observed on SD WebUI (using PyTorch) that different cuda versions of PyTorch get different results, and such results are larger or smaller depending on the model and prompt used, and I’m wondering if this difference is expected? Is there a way to reduce this difference?</p>
<p>Detailed contents here: <a href="https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/8257" rel="noopener nofollow ugc">Ensure consistency of results across different PyTorch versions · AUTOMATIC1111/stable-diffusion-webui · Discussion #8257 (github.com)</a></p>

username: ptrblck
cooked: <p>Could you describe the effects you are observing?<br>
Based on the cross-post it seems slightly different images are produced. Do you expect to get bitwise-identical results or is one output considered to be “wrong” for some reason?</p>

username: Shino
cooked: <p>According to my limited observation I found that the difference is mainly centered on the cuda version and has nothing to do with the pytorch version, I wonder why this difference exists, is it a bug? There is no error message during the running process.<br>
Second, whether this difference can be controlled, depending on the actual use situation, this may be very large, which causes trouble for the reproducible results and reduces the effect of some models.<br>
Finally, if evaluating these versions, I think 1.13.1+cu117 has the biggest problem, and many of the differences it produces will neither appear in 1.13.1+cu116 nor 2.0+cu118, but the latter is often closer.</p>

username: ptrblck
cooked: <p>You cannot expect to see bitwise-identical results when libraries are updated and small errors caused by the limited floating point precision are expected.<br>
It’s still unclear how large the errors are and if these are unexpected or not.</p>

username: Shino
cooked: <p>Is there an easy way to build an older CUDA version of PyTorch for use under Win?</p>

username: ptrblck
cooked: <p>It depends what your definition of “easy” is, but you should be able to build PyTorch from source on Windows using <a href="https://github.com/pytorch/pytorch#from-source">these instructions</a>. I’m not deeply familiar with the Windows build steps as I’m using Linux systems.</p>
Layernorm questions with Transformers

username: P-Sood
cooked: <p>So my current model has two transformers, (a and b), and we calculate the output from this a and b. For b we run a LayerNorm operation, then we concatenate to create ab. This is a late fusion concatenation model.</p>
<p>From ab we just run a Dropout and then a Linear layer to classify.</p>
<p>Now my model has started to overfit the train set and generalize poorly on the validation set.</p>
<p>Extra details: I have 8000 pieces of data for my train, with batch sizes of (1.2,4). Where I use a weighted random sampler and unweighted crossEntropyLoss to deal with class imbalance already.</p>
<p>Questions:</p>
<ol>
<li>
<p>Can the LayerNorm used for b also be run on a, or do I require separate Layernorm layers?</p>
</li>
<li>
<p>Now since I use LayerNorm before the concatenation, would it be fine if I just applied LayerNorm to ab, or rather the concatenation of a and b? Instead of creating two separate LayerNorms. or would the two separate outputs being concatenated first and then normed have a weird effect in losing understanding the single transformers both gave?</p>
</li>
<li>
<p>Should I even use LayerNorm at all?</p>
</li>
<li>
<p>For the classification head, is it right to be:<br>
Transformer outputs → Concatenation → LayerNorm → Dropout → Linear<br>
Currently, I am doing:<br>
Transformer outputs  → LayerNorm (only on b) → Concatenation → Dropout → Linear</p>
</li>
<li>
<p>Should I apply LayerNorm on the validation and test datasets?</p>
</li>
<li>
<p>My Current dropout is 0.2, would it generalize better to just use it at .5 or something greater than .2 for less overfitting?</p>
</li>
<li>
<p>Any tips to help reduce overfitting and increase generalizability would be welcome.</p>
</li>
</ol>
<p>Thank you in advance. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: P-Sood
cooked: <p>Any help or tips would be appreciated <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
[GPU Memory] Do I need to explicitly del local tensor that is copied to GPU?

username: Hui_Liu
cooked: <p>I am debugging a code that has increasing GPU memory when looping, the code is like this</p>
<pre><code class="lang-auto">  pbar = enumerate(dataloader)
.. 
  pbar = tqdm(pbar, total=nb)  # progress bar
  optimizer.zero_grad()
  for i, (imgs, targets, paths, _) in pbar: 
       imgs = imgs.to(device, non_blocking=True).float() / 255.0
       ....
      loss, loss_items = compute_loss_ota(pred, targets.to(device), imgs)
</code></pre>
<p>My questions are:</p>
<ol>
<li>
<p>Do I need to explicitly del <code>imgs</code> after each for loop to release memory? So if I understand correctly, for each iteration, imgs get a new batch of images on CPU, then imgs gets reassigned to the copy on GPU. Then, for the next iteration, imgs is assigned to a new batch of images, so the GPU copy of data got automatically deleted because there was no reference to it?</p>
</li>
<li>
<p>for GPU copy of targets to <code>compute_loss_ota</code>, it will be cleared automatically because it is a local variable?</p>
</li>
</ol>
<p>Thanks!</p>

username: ptrblck
cooked: <ol>
<li>Yes, <code>imgs</code> will be deleted unless you are explicitly storing a reference to the “old” <code>imgs</code> tensor somewhere.</li>
<li>Also yes, unless you store a reference to it (same as 1).</li>
</ol>
<p>Usually, an increase in memory usage is caused by storing the tensors unknowingly e.g. by accumulating them into a single tensor or by appending them to a <code>list</code> etc.</p>
How can Tensorboard open a profile trace > 2GB?

username: walle_autoscale
cooked: <p>Hi, I am having trouble opening a profile output file which is about 2GB. This profile only contains one iteration and cannot be smaller.<br>
I can see the <code>memory</code>, <code>kernels</code> page but I cannot open <code>trace</code>, it seems to be too big. Is there any solution?</p>
<p>this is ok:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e0958f0f2004d1de5396c1220b4c6a05c20b9550.png" data-download-href="https://discuss.pytorch.org/uploads/default/e0958f0f2004d1de5396c1220b4c6a05c20b9550" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e0958f0f2004d1de5396c1220b4c6a05c20b9550_2_690x279.png" alt="image" data-base62-sha1="w2LnFc4r7Qdq46UFiYSYzL74hbO" width="690" height="279" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e0958f0f2004d1de5396c1220b4c6a05c20b9550_2_690x279.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e0958f0f2004d1de5396c1220b4c6a05c20b9550_2_1035x418.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e0958f0f2004d1de5396c1220b4c6a05c20b9550_2_1380x558.png 2x" data-dominant-color="D3D7E2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2254×913 95.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>cannot open:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3cd06c717d0191863d56494216d5d76de6bf0561.png" data-download-href="https://discuss.pytorch.org/uploads/default/3cd06c717d0191863d56494216d5d76de6bf0561" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3cd06c717d0191863d56494216d5d76de6bf0561_2_690x191.png" alt="image" data-base62-sha1="8FZc0KOQtmhM7eYPGiToAcKxECR" width="690" height="191" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3cd06c717d0191863d56494216d5d76de6bf0561_2_690x191.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3cd06c717d0191863d56494216d5d76de6bf0561_2_1035x286.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3cd06c717d0191863d56494216d5d76de6bf0561_2_1380x382.png 2x" data-dominant-color="F7EADD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1520×421 8.03 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How can I use both system memory and gpu memory efficiently to speed up training?

username: Oliver
cooked: <p>Hi all,</p>
<p>I am using colab and sometimes kaggle instances and noticed that the GPU memory is often quite full but the system memory rarely runs above 4 gb and has a lot of space.</p>
<p>Is there some way that I can use the system memory more efficiently?</p>
<p>I am reading images from the disk, should I prefetch some and put them in memory?<br>
Is there a function that I am missing?<br>
Does prefetch do that (I understood from the github and the docs that it would prefetch into GPU memory not system memory, did I understand correctly?)?</p>
<p>Thank you all in advance for any tips you can offer!</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> will not move (or prefetch) data on the GPU by default and depends on the behavior implemented in the <code>Dataset.__getitem__</code> method. Usually you would load each sample into the host RAM and thus the <code>DataLoader</code>’s workers will also prefetch these batches on the host.<br>
Besides that you should note that moving data between the CPU and GPU can be quite expensive. While CPU offloading is available in PyTorch (which moves intermediates to the CPU to save GPU memory), this util. is used to allow the training of large models which would otherwise not fit into the GPU, which does not seem to be the case here.</p>

username: Oliver
cooked: <p>Thanks a lot for the explanation, I think I got it wrong the first time.<br>
So to make sure I understood correctly, I’m using <code>cv2.imread</code> (is there a benefit of using torchvisions native <code>read_image</code>?), I have set <code>pin_to_memory = True</code>, so the images I prefetch are in RAM and do not get loaded into the GPU queued up for processing?<br>
In this case, since I am only using a little RAM, increasing the <code>prefetch_factor</code> which I understand defaulting to <code>2</code>, could increase performance? I am going to try this.</p>

username: ptrblck
cooked: <p>Your understanding about the loading to the host RAM is correct.<br>
Increasing the <code>prefetch_factor</code> won’t necessarily improve the performance if the queue is already filled and the training is not consuming the prefetched batches fast enough.</p>

username: tiramisuNcustard
cooked: <p><code>pin_memory=True</code> allocates memory (page locked). This speeds up copy to GPU. See <a href="https://pytorch.org/docs/stable/notes/cuda.html#use-pinned-memory-buffers" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/cuda.html#use-pinned-memory-buffers</a></p>
Use part of resnet50

username: nikiguo93
cooked: <p>Hi, I want to use the output of the first block of resnet50. The output is of size 256<em>56</em>56. Then this would be fed into a relu layer and later a fully connected layer. I want to make the fc layer output features of size 1*256. but now I have error message like: RuntimeError: mat1 and mat2 shapes cannot be multiplied (14336x56 and 3136x256)<br>
can anybody tell me how to deal with the problem? I know pooling layer can help but is there any other options?</p>
Batch normalization different between .eval and .train modes even when running and batch statistics are the same

username: Nick_Halden
cooked: <p>Hey,<br>
I know batch normalization uses different statistics on eval() mode and train() mode, but, when<br>
I make those statistics the same, it still gives me different values.</p>
<p>Here’s the code to reproduce what I am talking about</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
import torch.nn as nn

a = torch.tensor([[[1,2],
                     [3,4]]]).float()

b = torch.tensor([[[10,20],
                     [30,40]]]).float()

X = torch.stack((a,b)).float() 
assert X.shape == (2,1,2,2)

l = nn.BatchNorm2d(1, momentum=1, eps=0).train() #1 channel
l(X) #calculate running_means and running_var

def batchnorm(x,u,var):
    return (x - u)/(torch.sqrt(var)) #epsilon is 0

one = batchnorm(X,l.running_mean, l.running_var)

# batch norm eval
l.eval()
two = l(X)

# batch norm train
l.train()
three = l(X)


assert (torch.abs(one - two) &lt; 0.0001).all()
assert not (torch.abs(one - three) &lt; 0.0001).all()
</code></pre>
<p>This code should run without problems because tensor <strong>one</strong> and <strong>three</strong> are different when I<br>
think they should be equal.</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>Your code returns the expected mismatches:</p>
<pre><code class="lang-python">torch.abs(one - two)
Out[15]: 
tensor([[[[5.9605e-08, 0.0000e+00],
          [5.9605e-08, 5.9605e-08]]],


        [[[0.0000e+00, 0.0000e+00],
          [0.0000e+00, 0.0000e+00]]]], grad_fn=&lt;AbsBackward0&gt;)

torch.abs(one - three)
Out[16]: 
tensor([[[[0.0598, 0.0551],
          [0.0504, 0.0457]]],


        [[[0.0176, 0.0293],
          [0.0762, 0.1231]]]], grad_fn=&lt;AbsBackward0&gt;)
</code></pre>
<p>The difference is expected as the <code>running_var</code> will be updated with Bessel’s correction as seen <a href="https://github.com/ptrblck/pytorch_misc/blob/31ac50c415f16cf7fec277dbdba72b9fb4d732d3/batch_norm_manual.py#L68-L69">here</a>.</p>
<pre><code class="lang-python">print(l.running_mean)
# tensor([13.7500])
print(X.mean([0, 2, 3]))
# tensor([13.7500])

print(l.running_var)
# tensor([216.7857])
print(X.var([0, 2, 3], unbiased=False))
# tensor([189.6875])
print(X.var([0, 2, 3], unbiased=False) * X.numel() / (X.numel() - 1))
# tensor([216.7857])
</code></pre>
Custom upsampling 4d tensor

username: core-not-dumped
cooked: <p>I created a 4D tensor of the following size. (B, C, H, W)<br>
And I want to reshape this tensor to the following size. (B, C, 2H, 2W)</p>
<p>Each value expands into 4 values, but only the elements in the original tensor that have an index corresponding to a value in the index tensor remain identical to the original tensor’s value.</p>
<p>The index follows the following rule.</p>
<p>0 1</p>
<p>2 3</p>
<p>And here is an example below:</p>
<pre data-code-wrap="python3"><code class="lang-plaintext">original tensor:
torch.Size([1, 1, 2, 2])
tensor([[[[1.0000, 0.4000],
          [0.2000, 0.5000]]]])

index tensor:
torch.Size([1, 1, 2, 2])
tensor([[[[0, 2],
          [1, 1]]]])
</code></pre>
<pre data-code-wrap="python3"><code class="lang-plaintext">output tensor:
torch.Size([1, 1, 4, 4])
tensor([[[[1.0000, 0.0000, 0.0000, 0.0000],
          [0.0000, 0.0000, 0.4000, 0.0000],
          [0.0000, 0.2000, 0.0000, 0.5000],
          [0.0000, 0.0000, 0.0000, 0.0000]]]])
</code></pre>
<p>How can I efficiently transform this tensor while making good utility of GPU? (maybe we can use torch.tensor.scatter_)</p>

username: KFrank
cooked: <p>Hi Core!</p>
<aside class="quote no-group" data-username="core-not-dumped" data-post="1" data-topic="174121" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/core-not-dumped/48/57944_2.png" class="avatar"> core-not-dumped:</div>
<blockquote>
<p>only the elements in the original tensor that have an index corresponding to a value in the index tensor remain identical to the original tensor’s value.</p>
</blockquote>
</aside>
<p>I don’t follow your upsampling algorithm.</p>
<p>Rather than give examples, could you post a (complete, runnable) script that<br>
performs the upsampling, using for-loops, etc., as necessary?</p>
<p>Best.</p>
<p>K. Frank</p>
Filling c++ arrays with each row of a 2D tensor

username: Wesley_Neill
cooked: <p>If I am trying to dump each row of a 2D tensor into an array in C++, is this appropriate?</p>
<pre><code class="lang-auto">array&lt;float, 3&gt; data;

float *values_ptr = &amp;tensor_data.data_ptr&lt;float&gt;()[i * tensor_data.stride(1)]
for (int i=0; i &lt; 3; i++) {
	data[i] = values_ptr[i];
}
</code></pre>
<p>Or is there a faster or more memory-efficient way to do this?</p>
Pretrained model to extract face or body position features

username: Gorgen
cooked: <p>my dataset is video from TV series “The Old Friends”. Could you please recommend pre-trained model to extract face or body position information ?</p>
<p>Thanks, best wishes<br>
please see the video as follow website.<br>
<a href="https://www.youtube.com/watch?v=AlGABMe4cmc" rel="noopener nofollow ugc">https://www.youtube.com/watch?v=AlGABMe4cmc</a></p>
What does torch.Tensor.normal_() actually do?

username: mariaalfaroc
cooked: <p>Hi!</p>
<p>Could any of you please help me understand what <code>torch.Tensor.normal_()</code> is actually doing? Because I’m running some test and I don’t understand why I’m getting two different outputs when I think they should be the same.</p>
<pre><code class="lang-auto">import torch
a = torch.rand(2, 2)
print(a)
# tensor([[0.8657, 0.5614],
        [0.7639, 0.0196]])

print(a.normal_())
# tensor([[-0.3112,  0.2799],
        [-1.8803,  0.0472]])

print(a.normal_(0, 1))
# tensor([[-1.4572, -0.1000],
        [-1.6295,  2.0425]])
</code></pre>

username: TzviNoy
cooked: <p>from the documentation:<br>
“Fills <code>self</code> tensor with elements samples from the normal distribution parameterized by <code>mean</code> and <code>std</code>”</p>
<p>Every time you run a.normal it will fill your tensor with new values from the normal distribution with the mean and std that you defined</p>

username: mariaalfaroc
cooked: <p>And the actual values from the original tensor are not used at all right?</p>

username: TzviNoy
cooked: <p>It seems like normal_ is overwriting the old values with the new ones.</p>
How to retrieve the loss function of FasterRCNN for object detection?

username: johnny69
cooked: <p>Context: I’m using the pretrained model <code>fasterrcnn_resnet50_fpn</code>. If the model is on train mode, it works like this:</p>
<pre><code class="lang-auto">model.train()
model(image, targets) # pass BOTH the image and the targets
</code></pre>
<pre><code class="lang-auto"># returns the loss
{'loss_classifier': tensor(2.0448, device='cuda:0', grad_fn=&lt;NllLossBackward0&gt;),
 'loss_box_reg': tensor(0.6015, device='cuda:0', grad_fn=&lt;DivBackward0&gt;),
 'loss_objectness': tensor(1.4119, device='cuda:0',
        grad_fn=&lt;BinaryCrossEntropyWithLogitsBackward0&gt;),
 'loss_rpn_box_reg': tensor(0.0668, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)}
</code></pre>
<p>While if the model is on eval mode, it works like this:</p>
<pre><code class="lang-auto">model.eval()
model(image) # pass ONLY the image
</code></pre>
<pre><code class="lang-auto"># Returns the predicted targets, but not the loss
{'boxes': tensor([[142.8750, 135.8750, 176.1250, 169.1250],
        [130.8750, 116.8750, 164.1250, 150.1250],
        [ 91.1250, 180.1250, 113.8750, 202.8750],
        [131.1250, 145.1250, 153.8750, 167.8750],
        [ 65.1250, 103.1250,  87.8750, 125.8750],
        [ -3.8750,  91.1250,  18.8750, 113.8750],
        [208.1250,  21.1250, 230.8750,  43.8750]]), 'labels': tensor([3, 3, 4, 4, 4, 4, 4])}
</code></pre>
<p>What I want to be able to do is find the <strong>loss function</strong> that <code>fasterrcnn_resnet50_fpn</code>, such that I could pass the predicted and actual targets into the function to get the losses.</p>
<p>e.g.</p>
<pre><code class="lang-auto">image, actual_targets = X_train[0]
model.eval()
pred_targets = model(image)

loss_fn(pred_targets, actual_targets)
</code></pre>
<p>Does anyone know how I could do this?</p>

username: ptrblck
cooked: <p>The losses are calculated <a href="https://github.com/pytorch/vision/blob/beb4bb706b5e13009cb5d5586505c6d2896d184a/torchvision/models/detection/generalized_rcnn.py#L104-L105">here</a> in the <code>GeneralizedRCNN.forward</code> method so you might be able to reimplement the <code>forward</code> method and pass the targets to during the validation pass, too.</p>

username: johnny69
cooked: <p>What I’m more looking for is a function to compare two sets of targets. I wouldn’t want to change the model as such, ideally I’d be able to run the loss function without running the model.</p>
<p>Is there any way I could do this instead?</p>
Where can I find at::clamp_min funciton?

username: core-not-dumped
cooked: <p>I found that the <code>torch.nn.functional.relu</code> function calls <code>torch.relu</code> and ‘torch.relu_’</p>
<pre><code class="lang-auto">def relu(input: Tensor, inplace: bool = False) -&gt; Tensor:
    r"""relu(input, inplace=False) -&gt; Tensor
    Applies the rectified linear unit function element-wise. See
    :class:`~torch.nn.ReLU` for more details.
    """
    if has_torch_function_unary(input):
        return handle_torch_function(relu, (input,), input, inplace=inplace)
    if inplace:
        result = torch.relu_(input)
    else:
        result = torch.relu(input)
    return result


relu_ = _add_docstr(
    torch.relu_,
    r"""
relu_(input) -&gt; Tensor
In-place version of :func:`~relu`.
""",
)
</code></pre>
<p>And I finally got this function <code>at::clamp_min(self, 0)</code>. But I’m not sure where it is mentioned on GitHub.</p>
<pre><code class="lang-auto">Tensor relu(const Tensor &amp; self) {
  TORCH_CHECK(self.scalar_type() != at::kBool, "Boolean inputs not supported for relu");
  return at::clamp_min(self, 0);
}

Tensor &amp; relu_(Tensor &amp; self) {
  TORCH_CHECK(self.scalar_type() != at::kBool, "Boolean inputs not supported for relu");
  return at::clamp_min_(self, 0);
}
</code></pre>
<p>Could you please provide me with the GitHub address where this function can be found?</p>

username: ptrblck
cooked: <p>The CPU kernel can be found <a href="https://github.com/pytorch/pytorch/blob/dcc159d3b660815ec0ee29ec3fc175b84cdf2967/aten/src/ATen/native/cpu/TensorCompareKernel.cpp#L332-L390">here</a> and the CUDA kernel <a href="https://github.com/pytorch/pytorch/blob/dcc159d3b660815ec0ee29ec3fc175b84cdf2967/aten/src/ATen/native/cuda/TensorCompare.cu#L42-L91">here</a>.</p>
Fine-tuning convolutional filters

username: sooh
cooked: <p>Hi, I want to fine-tune some of the convolutional filters in ResNet-50. To achieve this, I need to freeze the other convolutional filters. As far as I know, I can only freeze convolutional “layers”, but not convolutional “filters”.<br>
I already tried the following code, but it seems that it is impossible to make selected convolutional filters require gradient.</p>
<pre><code class="lang-auto">import torch
import torchvision

net = torchvision.models.resnet50(pretrained=True)
for layer_num, (name, param) in enumerate(net.named_parameters()):
    if 'conv' in name:
        param.requires_grad = False
        param[0].requires_grad = True
        print(param.requires_grad, param[0].requires_grad) # expected False True, but False False
</code></pre>
<p>Is there anyway to perform this kind of fine-tuning?</p>

username: J_Johnson
cooked: <aside class="quote no-group" data-username="sooh" data-post="1" data-topic="174105">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e36b37/48.png" class="avatar"> sooh:</div>
<blockquote>
<p>As far as I know, I can only freeze convolutional “layers”, but not convolutional “filters”.</p>
</blockquote>
</aside>
<p>When you say, “filters” and “layers”, do you mean weights and biases? If that is the case, here is one way you might go about it:</p>
<pre><code class="lang-auto">import torchvision
import torch.nn as nn

net = torchvision.models.resnet50(pretrained=True)
with torch.no_grad():
    for layer in net.modules():
        if isinstance(layer, nn.Conv2d):
            print(layer)
            print(layer.weight.requires_grad)
            layer.weight.requires_grad = False
            print(layer.weight.requires_grad) 
            if not isinstance(layer.bias, type(None)): #Check if layer has no bias, otherwise you'll get an error that NoneType object has no requires_grad
                print(layer.bias.requires_grad)
</code></pre>
<p>As a side note, the Conv2d layers in the ResNet-50 do not use bias because they use BatchNorm2d, and a bias would be redundant. To target those, just use:</p>
<pre><code class="lang-auto">if isinstance(layer, nn.BatchNorm2d):
    ...
</code></pre>
Concatenation of losses raise an error

username: qm-intel
cooked: <p>I have two tensors</p>
<pre><code class="lang-auto"># losses_q
tensor(0.0870, device='cuda:0', grad_fn=&lt;SumBackward0&gt;)
# this_loss_q
tensor([0.0874], device='cuda:0', grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>when I am trying to concat them, pytorch raises an error:</p>
<pre><code class="lang-auto">losses_q = torch.cat((losses_q, this_loss_q), dim=0)

RuntimeError: zero-dimensional tensor (at position 0) cannot be concatenated
</code></pre>
<p>How to resolve this?</p>

username: ptrblck
cooked: <p><code>unsqueeze</code> should work as it will add a dimension to the tensor:</p>
<pre><code class="lang-python">losses_q = torch.cat((losses_q.unsqueeze(0), this_loss_q), dim=0)
</code></pre>

username: qm-intel
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks a lot, it worked.</p>
Install -U --pre DOWNGRADES my cuDNN instae of upgrading them

username: aifartist
cooked: <p>I want to upgrade the version of cuDNN from 8.5 to 8.8 in my PyTorch venv.<br>
cmd&gt; pip3 list | egrep cudnn<br>
nvidia-cudnn-cu11        8.5.0.96<br>
cmd&gt; pip3 install -U nvidia-cudnn-cu11<br>
cmd&gt; pip3 list | egrep cudnn<br>
nvidia-cudnn-cu11        2022.5.19<br>
nvidia-cudnn-cu116       8.4.0.27</p>
<p>Now I have my nvidia-cudnn-cu11’s version change to 2022.5.19, whatever that mean, and it also installed and even older cudnn based on cuda-11.6.</p>

username: ptrblck
cooked: <p>The PyTorch wheels ship with their specified dependencies and use the <code>RPATH</code> to link against needed libraries. If you want to update specific libraries, you should build from source instead with a locally installed CUDA toolkit and cuDNN libs.</p>
Save a trained (from a pre-trained) model and make inference from saved model

username: Schroter
cooked: <p>Hi All,</p>
<p>I am looking to save a pre-trained model after using transfer learning to train it on a seperate dataset. Then use the trained model to make predictions on test data. My current code is as below.</p>
<pre><code class="lang-auto">model_conv = torchvision.models.regnet_y_32gf(weights = 'RegNet_Y_32GF_Weights.IMAGENET1K_SWAG_E2E_V1')
#model_conv = torchvision.models.efficientnet_b7(weights = 'EfficientNet_B7_Weights.IMAGENET1K_V1')

for param in model_conv.parameters():
    param.requires_grad = False

# Parameters of newly constructed modules have requires_grad=True by default
num_ftrs = model_conv.fc.in_features
model_conv.fc = nn.Linear(num_ftrs, 2)

model_conv = model_conv.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that only parameters of final layer are being optimized as
# opposed to before.
optimizer_conv = optim.SGD(model_conv.fc.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)

model_conv = train_model(model_conv, criterion, optimizer_conv,
                         exp_lr_scheduler, num_epochs=1)

# Specify a path
PATH = "state_dict_model.pt"

# Save
torch.save(model_conv.state_dict(), PATH)

# Load
model = model_conv()
model.load_state_dict(torch.load(PATH))
model.eval()
</code></pre>
<p>Error for the above code:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_27/3952178397.py in &lt;module&gt;
      6 
      7 # Load
----&gt; 8 model = model_conv()
      9 model.load_state_dict(torch.load(PATH))
     10 model.eval()

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1188         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189                 or _global_forward_hooks or _global_forward_pre_hooks):
-&gt; 1190             return forward_call(*input, **kwargs)
   1191         # Do not call functions when jit is used
   1192         full_backward_hooks, non_full_backward_hooks = [], []

TypeError: forward() missing 1 required positional argument: 'x'
</code></pre>
<p>Would anyone be able to please help me in this regards.</p>
<p>Thanks &amp; Best Regards<br>
AMJS</p>

username: ptrblck
cooked: <p><code>model_conv</code> is already an instance of <code>torchvision.models.regnet_y_32gf(weights = 'RegNet_Y_32GF_Weights.IMAGENET1K_SWAG_E2E_V1')</code> to calling it would execute the forward pass.<br>
In your code snippet it seems you want to initialize the <code>model</code> variable with <code>model_conv</code>, which is wrong and you can directly use the <code>model_conv</code> object instead.</p>