akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is 3080 not compatible with torch2.1?

username: Jachilles
cooked: <p>Is 3080 not compatiable with torch2.1?</p>
<p>I am using Ubuntu20.04 with nvidia 3080 rtx gpus.</p>
<p>I’ve made a docker image by using <code>nvidia/cuda:11.8.0-devel-ubuntu20.04</code>, and I installed torch2.1. But the installed torch does not detect a nvidia device.</p>

username: ptrblck
cooked: <p>All currently built PyTorch binaries support your 3080. Make sure the right binary was installed by checking <code>torch.version.cuda</code>. If this returns the right CUDA version you would need to check your setup and make sure applications can communicate with your GPU via a properly installed NVIDIA driver.</p>
Why does the triu_tril_kernel need to compute self_offset and result_offset

username: ipe-zhangyz
cooked: <p>The CUDA backend kernel of torch.tril and torch.triu triu_tril_kernel: <a href="https://github.com/pytorch/pytorch/blob/871e27a61c0f0bf49dab51f68534c49c95ee0ee3/aten/src/ATen/native/cuda/TriangularOps.cu#L28C27-L28C27" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/871e27a61c0f0bf49dab51f68534c49c95ee0ee3/aten/src/ATen/native/cuda/TriangularOps.cu#L28C27-L28C27</a> computes <em>self_offset</em> and <em>result_offset</em> explicitly. Seems that the two variable equal to <em>linear_idx</em> provided that <em>linear_idx</em> is less than the total element number of the tensor. In which case will the two differ from <em>linear_idx</em>? Thanks!<br>
Here is the code:</p>
<pre><code class="lang-auto">template &lt;typename scalar_t, typename IndexType, bool upper&gt;
C10_LAUNCH_BOUNDS_1(cuda::getApplyBlockSize())
__global__ void triu_tril_kernel(
    cuda::detail::TensorInfo&lt;scalar_t, IndexType&gt; result_info,
    const cuda::detail::TensorInfo&lt;scalar_t, IndexType&gt; self_info,
    const int64_t k,
    const int64_t N) {
  int64_t linear_idx = blockIdx.x * blockDim.x + threadIdx.x;
  if (linear_idx &gt;= N) {
    return;
  }

  auto dims = self_info.dims;

  IndexType self_offset = 0, result_offset = 0;
  // Compute column index and corresponding offset
  IndexType col = linear_idx % self_info.sizes[dims - 1];
  linear_idx /= self_info.sizes[dims - 1];
  self_offset += self_info.strides[dims - 1] * col;
  result_offset += result_info.strides[dims - 1] * col;

  // Compute row index and corresponding offset
  IndexType row = linear_idx % self_info.sizes[dims - 2];
  linear_idx /= self_info.sizes[dims - 2];
  self_offset += self_info.strides[dims - 2] * row;
  result_offset += result_info.strides[dims - 2] * row;

  // Compute remaining offsets
  IndexType running_index;
  #pragma unroll
  for (IndexType i = dims - 3; i &gt;= 0; --i) {
    running_index = linear_idx % self_info.sizes[i];
    linear_idx /= self_info.sizes[i];
    self_offset += running_index * self_info.strides[i];
    result_offset += running_index * result_info.strides[i];
  }

  bool mask = upper ? (col - row &gt;= k) : (col - row &lt;= k);
  result_info.data[result_offset] = mask ? self_info.data[self_offset] : scalar_t(0);
}
</code></pre>
Is the use of model.eval() correct?

username: Xuei
cooked: <p>I’m trying to use PyTorch to build my model.<br>
I use <code>model.eval()</code> for validation data and also for test data.</p>
<pre><code class="lang-auto">if mode == "train":
        model.train()
    elif mode == "test":
        model.eval()
    cost = correct = 0
    for feature, target in tqdm(dataloader, desc=mode.title()):
        feature, target = feature.to(device), target.to(device)
        output = model(feature)
        loss = criterion(output, target)
        
        if mode == "train":
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        
        cost += loss.item() * feature.shape[0]
        correct += (output.argmax(1) == target).sum().item()
    cost = cost / len(dataset)
    acc = correct / len(dataset)
    return cost, acc
</code></pre>
<pre><code class="lang-auto">#test eval
def eval_model(model, testloader, criterion, device):
    model.load_state_dict(torch.load('/model/best_model.pth'))
    model.eval()
    total_loss = 0.0
    all_labels = []
    all_predictions = []

    with torch.no_grad():
        for inputs, labels in testloader:
            inputs, labels = inputs.to(device), labels.to(device)
            outputs = model(inputs)
            loss = criterion(outputs, labels)
            total_loss += loss.item()

            _, predictions = torch.max(outputs, 1)
            all_labels.extend(labels.cpu().numpy())
            all_predictions.extend(predictions.cpu().numpy())

    average_loss = total_loss / len(testloader)
    accuracy = sum(np.array(all_labels) == np.array(all_predictions)) / len(all_labels)
</code></pre>
<p>Is this correct?<br>
If there’s still something wrong, please help me fix it. Thank you</p>

username: ptrblck
cooked: <p>Your code looks correct assuming you are passing the right <code>mode</code> to this function.</p>

username: Xuei
cooked: <p>ok,<br>
Thank you for your help.</p>
Is scaled_dot_product_attention included in TransformerEncoder

username: yuzhenmao
cooked: <p>Hi,</p>
<p>From <a href="https://pytorch.org/tutorials/intermediate/scaled_dot_product_attention_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">(Beta) Implementing High-Performance Transformers with Scaled Dot Product Attention (SDPA) — PyTorch Tutorials 2.2.0+cu121 documentation</a> it mentions that  “This function (scaled_dot_product_attention) has already been incorporated into <code>torch.nn.MultiheadAttention</code> and <code>torch.nn.TransformerEncoderLayer</code>”. However, in the doc of TransformerEncoder, it lists several constraints that have to be satisfied if we want to use flash-atten (It seems that TransformerEncoder can only use flash-atten during the inference time):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9da46f811dc3e0f5924aa086d7aa5c9cdd3e6868.png" data-download-href="https://discuss.pytorch.org/uploads/default/9da46f811dc3e0f5924aa086d7aa5c9cdd3e6868" title="Screenshot 2023-11-02 at 4.44.18 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9da46f811dc3e0f5924aa086d7aa5c9cdd3e6868_2_690x341.png" alt="Screenshot 2023-11-02 at 4.44.18 PM" data-base62-sha1="muzgotoRWurm4rqVwmbOJKIH1Lq" width="690" height="341" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9da46f811dc3e0f5924aa086d7aa5c9cdd3e6868_2_690x341.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9da46f811dc3e0f5924aa086d7aa5c9cdd3e6868_2_1035x511.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9da46f811dc3e0f5924aa086d7aa5c9cdd3e6868_2_1380x682.png 2x" data-dominant-color="F1F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-11-02 at 4.44.18 PM</span><span class="informations">1472×728 98.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But scaled_dot_product_attention does not mention such constraints in its doc. Is this inconsistency correct, or I misunderstand something? Thanks!</p>
Small bug in torchrl

username: Jean-Marc_Fauche
cooked: <p>I noticed a small bug in torchrl.envs.utils:<br>
Indeed in the _per_level_env_check function, line 394 it is coded</p>
<pre><code class="lang-auto">if _data0.shape != _data1.shape:
                raise AssertionError(
                    f"The shapes of the real and fake tensordict don't match for key {key}. "
                    f"Got fake={_data0.shape} and real={_data0.shape}."
                )
</code></pre>
<p>of course it should be:</p>
<pre><code class="lang-auto">if _data0.shape != _data1.shape:
                raise AssertionError(
                    f"The shapes of the real and fake tensordict don't match for key {key}. "
                    f"Got fake={_data0.shape} and real={_data1.shape}."
                )
</code></pre>
<p>This obviously has no impact on operation but can be confusing for those who encounter the error message… (of which I am one <img src="https://discuss.pytorch.org/images/emoji/apple/exploding_head.png?v=12" title=":exploding_head:" class="emoji" alt=":exploding_head:" loading="lazy" width="20" height="20">)</p>

username: ptrblck
cooked: <p>Thanks for reporting! Would you be interested in submitting a fix to <code>torchrl</code>?</p>

username: Jean-Marc_Fauche
cooked: <p>How do you do this? (That said, changing a 0 to 1 is not really complicated…)</p>

username: ptrblck
cooked: <p>You can check the <a href="https://pytorch.org/docs/stable/community/contribution_guide.html">Contribution Guide</a> and create the fix in <a href="https://github.com/pytorch/rl"><code>pytorch/rl</code></a>. The code owners will then take a look at the fix and merge it.</p>
How to get step time from torch.profiler?

username: yoko
cooked: <p>I would like to get time for a step from torch profiler. Is there any way to do?</p>
<pre><code class="lang-python">skip_first = 5
wait = 5
warmup = 5
active = 5
repeat = 3

num_steps = wait + warmup + active
iterations = skip_first + num_steps * repeat

with torch.profiler.profile(
            activities=[
                torch.profiler.ProfilerActivity.CPU,
                torch.profiler.ProfilerActivity.CUDA,
            ],
            schedule=torch.profiler.schedule(
                skip_first=skip_first,
                wait=wait,
                warmup=warmup,
                active=active,
                repeat=repeat,
            ),
            on_trace_ready=torch.profiler.tensorboard_trace_handler("./log"),
        ) as p:
            for iter in range(iterations):
                model(input)
                p.step()
print(p.key_averages().total_average().self_cuda_time_total)
</code></pre>
A gpu out of memory error occurred while moving the tensor to the CPU

username: 111220
cooked: <p>During the calculation of proposal and target iou in detectron2 framework, if OOM appeared, it would try to transfer it to CPU for calculation, but the error of gpu out of memory still occurred when moving to CPU</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "main.py", line 76, in &lt;module&gt;
    loss_dict = model(data)
  File "/root/anaconda3/envs/python367/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/detectron2/detectron2/modeling/meta_arch/rcnn.py", line 124, in forward
    proposals, proposal_losses = self.proposal_generator(images, features, gt_instances)
  File "/root/anaconda3/envs/python367/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/yaogan/custom/kaggle_rpn.py", line 114, in forward
    losses = outputs.losses()
  File "/root/detectron2/detectron2/modeling/proposal_generator/rpn_outputs.py", line 333, in losses
    gt_objectness_logits, gt_anchor_deltas = self._get_ground_truth()
  File "/root/yaogan/custom/kaggle_rpn.py", line 57, in _get_ground_truth
    matched_idxs, gt_objectness_logits_i = retry_if_cuda_oom(self.anchor_matcher)(match_quality_matrix)
  File "/root/detectron2/detectron2/utils/memory.py", line 84, in wrapped
    return func(*new_args, **new_kwargs)
  File "/root/detectron2/detectron2/utils/memory.py", line 82, in &lt;genexpr&gt;
    new_args = (maybe_to_cpu(x) for x in args)
  File "/root/detectron2/detectron2/utils/memory.py", line 65, in maybe_to_cpu
    return x.to(device="cpu")
RuntimeError: CUDA out of memory. Tried to allocate 2.27 GiB (GPU 0; 10.76 GiB total capacity; 7.76 GiB already allocated; 334.69 MiB free; 9.60 GiB reserved in total by PyTorch)

</code></pre>
<p>Why ?</p>

username: maxencewynen
cooked: <p>HI, have you found out the cause to this problem? I’m facing a similar issue</p>
Convert fasterrcnn_resnet50_fpn to torchscript

username: sakh251
cooked: <p>I fine tuned a fasterrcnn_resnet50_fpn model and try to convert it to torchscript.</p>
<p>Load pre-trained model:</p>
<p><code>model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)</code></p>
<p>After training:</p>
<pre><code class="lang-auto">in_size = 416
inp = torch.Tensor(np.random.uniform(0.0, 250.0, size=(1, 3, in_size, in_size)))
inp = inp.cuda()
with torch.no_grad():
    out = model(inp)
    model_trace = torch.jit.trace(model, inp)
</code></pre>
<p>I got this error:</p>
<pre><code class="lang-auto">RuntimeError: Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions
</code></pre>
<p>Thank you for any help.<br>
I do not want use it on mobile devices.</p>

username: ptrblck
cooked: <p>You could try to use <code>torch.jit.script</code> instead, which should be more flexible.</p>

username: sakh251
cooked: <p>Thank you. It works.<br>
not sure about the differences?</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/docs/stable/jit.html#mixing-tracing-and-scripting">This section</a> of the docs explain the main differences.</p>

username: sakh251
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you for your reply.<br>
I can convert it to torchscript in script mode.<br>
However I need to have it in traced mode. I am converting it by AWS neo which is based on Apache TVM.<br>
So I need to solve this error.</p>
<pre><code class="lang-auto">Only tensors, lists, tuples of tensors, or dictionary of tensors can be output from traced functions
</code></pre>
<p>Thank you for any clue or solution.</p>

username: Vidit_Agarwal
cooked: <p>You are getting this error because your model will be giving an output which is not <code>tensors, lists, tuples of tensors or dictionary of tensors</code>.<br>
This can be fixed by modifying the forward function of your model class. Just wrap the current output with <code>torch.tensor($current_output$)</code>.</p>
Does torch still use GPU when torch.backends.cudnn.enabled=False?

username: buttercroissant
cooked: <p>Or are computations offloaded to CPU? I noticed that my CPU utilization jumped from 10-20% to 100% after toggling torch.backends.cudnn.enabled from true to false.</p>

username: ptrblck
cooked: <p>Yes, PyTorch will use native CUDA kernels if cuDNN is disabled.</p>

username: buttercroissant
cooked: <p>Thanks so much for the fast reply!! Do you have any insight into why setting torch.backends.cudnn.enabled to false would drastically increase CPU utlilization like this?</p>

username: ptrblck
cooked: <p>No, but you could profile the workload to check where the increased CPU workload is coming from.</p>
Torchaudio Spectrogram Usage

username: cinjon
cooked: <p>I am using the MelSpectrogram, which is just using Spectrogram under the hood, from torchaudio. I am also using a Conformer from torchaudio. When I put the results of the spectrogram transform into the Conformer, it appears that I need to transpose it. Is that right? I’m surprised if everyone is doing that…</p>
<pre><code class="lang-auto">        self.transform = torchaudio.transforms.MelSpectrogram(
            sample_rate=config.sample_rate,
            n_fft=n_fft,
            hop_length=hop_length,
            n_mels=n_mels
        )   
        self.conformer = torchaudio.models.Conformer(
            input_dim=config.input_dim,
            num_heads=config.num_heads,
            ffn_dim=config.ffn_dim,
            num_layers=config.num_layers,
            depthwise_conv_kernel_size=config.depthwise_conv_kernel_size,
            dropout=config.dropout,
            use_group_norm=config.use_group_norm,
            convolution_first=config.convolution_first,
        )

            mel_spectrogram = self.transform(audio)
            mel_spectrogram = torch.transpose(mel_spectrogram, 1, 2)
            log_mel = torch.log(mel_spectrogram + 1e-9)
            bs, length, _ = log_mel.shape
            lengths = torch.ones(bs, device=log_mel.device) * length
            output, _ = self.conformer(log_mel, lengths)
</code></pre>
Difference between `torch.dtype` and `torch.tensortype`

username: Guanqun_Yang
cooked: <h1>Problem</h1>
<p>Recently I ran into an error when I tried to convert an array into float tensor using <code>torch.tensor(arr, dtype=torch.FloatTensor)</code>.</p>
<pre><code class="lang-auto">TypeError: tensor(): argument 'dtype' must be torch.dtype, not torch.tensortype
</code></pre>
<p>I checked out the documentation <a href="https://pytorch.org/docs/stable/tensor_attributes.html#tensor-attributes" rel="nofollow noopener">here</a> and it turns out I did not have correct understanding about data type. I previously thought <code>torch.float</code> is equivalent to <code>torch.FloatTensor</code>.</p>
<p>However, I still do not get the difference between those two and rationale behind this design, which appears to be redundant at first glimpse. Maybe <code>torch.FloatTensor</code> depends on particular storage type whereas <code>torch.float</code> doe not?</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>dtype</code> is a datatype, like <code>torch.float</code> or <code>torch.double</code><br>
<code>tensortype</code> is a type of tensor, like <code>torch.FloatTensor</code>, <code>torch.DoubleTensor</code>.</p>

username: Ujjwal_Jain
cooked: <p>This below two code does the same thing. You can either explicitly provide the datatype or use that type of tensor.</p>
<p><code>float_tensor = torch.tensor([4,5,6], dtype=torch.float)</code></p>
<p><code>float_tensor = torch.FloatTensor([4,5,6])</code></p>

username: Aj-esh
cooked: <p>dtype is how actually pytorch call data type within itself when we try create a tensor.<br>
When we check the documentation, it says there, dtype=dtype,</p>
<p>When we check the documentation of torch.FloatTensor<br>
It is actually is torch.tensor([array], dtype= torch.float)</p>
<p>Here, internally, pytorch refers torch.float as dtype and for ease, they also created another function to create tensor of float.</p>
Using torch.autograder.grad on a multi-GPU setup

username: Richy
cooked: <p>Hey all, Does torch.autograd.grad have issues with a multi GPU setup I am getting this error below.</p>
<pre><code>grads = torch.autograd.grad(loss, parameters, allow_unused=allow_unused)
</code></pre>
<p>File “/opt/conda/envs/pytorch/lib/python3.10/site-packages/torch/autograd/<strong>init</strong>.py”, line 394, in grad<br>
result = Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.</p>
Model performance very different on mac with Intel and apple chips

username: gojiita_ku
cooked: <p>I ran the same code (with the same random seed) for training a resnet18 model for image classification on one Macbook with Intel chip and one Macbook with M1 chip, but got very different performance in terms of loss and accuracy. For example, the model achieved the best accuracy of 94% on the M1 chip but only obtained the best accuracy of 79% on the intel chip. The initial model parameters are exactly same on these two devices. Software (e.g. PyTorch, Python) versions are also same on two devices. Does anyone know what is causing this?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/6455175132727d4af5989e3f1894df978626d433.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6455175132727d4af5989e3f1894df978626d433" title="img"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/6455175132727d4af5989e3f1894df978626d433_2_690x260.jpeg" alt="img" data-base62-sha1="ejA3kZGyCDQ7jsiBiHzXAUsfTHB" width="690" height="260" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/6455175132727d4af5989e3f1894df978626d433_2_690x260.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/6455175132727d4af5989e3f1894df978626d433_2_1035x390.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/6455175132727d4af5989e3f1894df978626d433_2_1380x520.jpeg 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">img</span><span class="informations">2152×812 123 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Forward hooks break when model is deep copied

username: fralupo
cooked: <p>In order to extract intermediate features from a torchvision model, I implemented a class where the forward return the ouput of the network and the intermediate features.</p>
<p>This is what it looks like:</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
import torch.nn as nn
from copy import deepcopy

class CustomMobilenetV3(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = models.mobilenet_v3_small()
        self.features = []
        self.layers  = [3,8,11]
        for i,ln in enumerate(self.layers):
            self.model.features[ln].register_forward_hook(self.get_features())



    def get_features(self):
        def hook(model, input, output):
            self.features.append(output)
        return hook

    def forward(self, x):

        x = self.model(x)

        return x, self.features
</code></pre>
<p>This works fine If I do this:</p>
<pre><code class="lang-auto">device = torch.device('cpu')
x = torch.randn(1,3,192,192).to(device)

mymodel = CustomMobilenetV3()
o, features = mymodel(x)

for i in features:
    print(i.shape)
</code></pre>
<p>It does not work If I deepcopy the model, then run it:</p>
<pre><code class="lang-auto">deep_model = deepcopy(mymodel)

o, features = deep_model(x)
print(features)
for i in features:
    print(i.shape)
</code></pre>
<p>in this case the variable features remains empty.</p>
<p>Interestingly, this works fine:</p>
<pre><code class="lang-auto">def get_features():
    def hook(model, input, output):
        feats.append(output)
    return hook

model = models.mobilenet_v3_small()
layers  = [3,8,11]
for i,ln in enumerate(layers):
    model.features[ln].register_forward_hook(get_features())

deep_model = deepcopy(model)

feats = []

o = deep_model(x)
for f in feats:
    print(f.shape)
</code></pre>
<p>What am I missing?<br>
I need to deepcopy the model to use EMA.</p>
<p>Thanks!</p>

username: mmattia
cooked: <p>I am facing the same issue, have you found a solution to this?</p>
Number of function evaluations/gradient by torch.optim

username: Joaquin_Rapela
cooked: <p>Hello,</p>
<p>I am using <code>torch.optim.LBFGS</code> and I want to get the number of function evaluations performed during an optimization. Currently I am using:</p>
<blockquote>
<p>optimizer = torch.optim.LBFGS(x, **optim_params)<br>
stateOneEpoch = optimizer.state[optimizer._params[0]]<br>
nfeval = stateOneEpoch[“func_evals”]</p>
</blockquote>
<p>which gives a reasonable number. However, this seems to be an undocumented hack. Is there a standard, and documented, way of otaining the number of function evaluations?</p>
<p>How can I get the number of gradient evaluations?</p>
<p>Thanks, Joaquin</p>
Check if MPS is available before importing torch

username: gernophil
cooked: <p>Hey everyone,<br>
I have a hen and egg problem with <code>import torch</code>. I need to set the environmental variable <code>os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = str(1)</code>, before <code>import torch</code> otherwise it will have no effect. However, I only need to set it if <code>torch.backends.mps.is_available()</code> is <code>TRUE</code>. But to check this, I would have needed to have importetd torch before for what I would have needed to know, if MPS would be available. You see my problem? Any easy solution for that?</p>

username: ptrblck
cooked: <p>You could set the env variable globally since it won’t have any effect of MPS is not available.</p>
Killed, OOM but weird

username: helixing99
cooked: <p>Dear everyone,</p>
<p>Something weird happened: My code get killed after iterations.<br>
The error message is “killed”</p>
<p>I understand that there are many discussions about it, however, their solutions can’t apply to my case.</p>
<ol>
<li>It is not CUDA OOM.</li>
<li>I use<br>
cpu = psutil.cpu_percent()<br>
mem = psutil.virtual_memory().available * 100 / psutil.virtual_memory().total<br>
to check my memory usage, however, the percentage of available memory is always 60% until it is killed.</li>
<li>This machine is used by multiple people whose code and conda environments are definitely different (our code worked before), however, we all receive the same errors.</li>
<li>Except python code being killed, the machine can work smoothly without problems.</li>
</ol>
<p>The error log from dmesg is “Out of memory: Killed process 54477 (python) total-vm:61672084kB, anon-rss:32929660kB, file-rss:66904kB, shmem-rss:16228kB, UID:1011 pgtables:72632kB oom_score_adj:0”</p>
<p>I doubt there is problems with cuda or torch, however, since the machine is using by multiple people, it is hard to believe we all install the same conflicting package.</p>
<p>Appreciate all help or advice!!</p>
<p>Update:<br>
I reinstall cuda 11.8 and corrsponding torch but there is no luck.</p>

username: ptrblck
cooked: <p>Your are running out of host RAM and the OS kills your application before it dies.</p>
How to implement high dimensional grid sample?

username: DapengFeng
cooked: <p>Hi. I want to add a high-dimensional grid sample, e.g., 10D, but I do not know which files should be modified. Any Tips are appreciated.</p>

username: ptrblck
cooked: <p>I don’t fully understand the question so could you describe which files you want to modify in the first place?<br>
Do you want to implement a new method into PyTorch?</p>

username: DapengFeng
cooked: <p>Hi, ptrblck. I want to implement an arbitrary dimensional grid sampler within PyTorch. However, PyTorch only implements a 2D/3D grid sampler. I think some C++/CUDA codes in <code>GridSampler.xxx</code> located in <code>aten/src/ATen/native/cuda</code> should be modified, but I do not know whether other files in other folders should be modified?</p>

username: ptrblck
cooked: <p>Yes, the <code>GridSampler.cpp</code> file looks like the correct place to add new functionality.<br>
It also seems you want to add your new methods into the core framework.<br>
If so, you should create a feature request on GitHub and discuss it with the code owners (unless you only want to add it locally).<br>
An easier approach might be to create a custom CUDA extension to provide the new functionality.</p>

username: DapengFeng
cooked: <p>Thanks for your great help.</p>
() operator overloading in DispatchStub class

username: jiangxiaobin96
cooked: <p><code>build/aten/src/ATen/UfuncCPU_add.cpp</code><br>
After macro expansion and get</p>
<pre><code class="lang-c++">void structured_ufunc_add_CPU::impl(const at::Tensor &amp; self, const at::Tensor &amp; other, const at::Scalar &amp; alpha, const at::Tensor &amp; out) {
      add_stub(device_type(), *this, alpha);
}
</code></pre>
<p>I can not understand the value <code>add_stub()</code> here, because of <code>()operator overloading</code> in DispatchStub class, here add_stub can call following function, why does operator overloading have a return value but add_stub does’t have?</p>
<pre><code class="lang-c++">rT operator()(c10::DeviceType device_type, ArgTypes&amp;&amp;... args) {
    FnPtr call_ptr = get_call_ptr(device_type);
    return (*call_ptr)(std::forward&lt;ArgTypes&gt;(args)...);
  }
</code></pre>
Dtensor dynamic gather local tensor

username: HaitaoX
cooked: <p>If I create two matrix A,B based on  <strong>devicemesh [[0,1,2],[3,4,5],[6,7,8]] and dtensor</strong>, and I want to gather data from device_mesh’s subprocess group to the current process arbitrarily(for example only gather from ranks[1,3]), should I use <strong>dist.new_group</strong> or other methods?<strong>for example, I want to gather dA.local_tensor() from ranks=[1,4] and dB.local_tensor() from ranks=[3,4]</strong>,I need some help…I’ll loop through the modifications to gather’s sub-inbound groups, they’re randomized QAQ</p>
<pre><code class="lang-auto">@spawn_threads_and_init_comms
def collect_dtensor(world_size):
  rank = dist.get_rank()
  x = torch.arange(0,16).reshape(4,4)
  mesh = DeviceMesh("cpu", [[0,1],[2,3]],mesh_dim_names=['dp','tp'])
  dx = distribute_tensor(x, mesh,[Shard(0),Shard(1)])
  row_data = funcol.all_gather_tensor(dx.to_local(), gather_dim=1, group=(mesh, 1))
  dist.barrier()
  print(f"rank:{rank},{row_data}")

WORLD_SIZE= 4
collect_dtensor(WORLD_SIZE)
</code></pre>

username: HaitaoX
cooked: <p>solution:<a href="https://github.com/pytorch/pytorch/issues/112578" class="inline-onebox" rel="noopener nofollow ugc">Get the specified pg dtensor and devicemesh of given the ranks · Issue #112578 · pytorch/pytorch · GitHub</a></p>
Does torch.cuda.amp support O2 almost FP16 training now?

username: Kungbohan
cooked: <p>Hello,</p>
<p>When I use apex amp, it always shows a warning:<br>
Warning: apex.amp is deprecated and will be removed by the end of February 2023. Use [PyTorch AMP](<a href="https://pytorch.org/docs/stable/amp.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic Mixed Precision package - torch.amp — PyTorch 2.1 documentation</a><br>
Thus, I am trying to update my code (from apex to pytorch build-in).<br>
When I used apex library, my code adopts “O2” mixed precision, aka almost FP16 level.<br>
However, I did not see anything similar in the pytorch build-in module torch.autocase.<br>
There are only two types: torch.float16 and torch.bfloat16.<br>
Does this mean that pytorch did not implement “O2” mixed precision? Why?</p>
<p>On the other hand, I am also curious about the training speed.<br>
torch.float16 &gt; torch.bfloat16 &gt; O1 &gt; O0 &gt; O3 &gt; O2.<br>
Why O2 is the slowest?<br>
Note: O0: FP32, O1: Mixed Precision; O2: almost FP16 ; O3: FP16</p>
<p>Thank you very much!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Kungbohan" data-post="1" data-topic="191094">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kungbohan/48/64953_2.png" class="avatar"> Kungbohan:</div>
<blockquote>
<p>Does this mean that pytorch did not implement “O2” mixed precision? Why?</p>
</blockquote>
</aside>
<p>Yes, that’s correct as its deprecated <code>apex</code> implementation was too limiting in its flexibility and disabled a few important use cases. There are workarounds using a custom <code>optimizer</code> now holding the states.</p>
What is CompositeNodes of torch.profile?

username: yoko
cooked: <p>I use torch.profiler. What is CompositeNodes?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f614d819f8ffe89dc9137735a1936dcbc2cd170f.png" data-download-href="https://discuss.pytorch.org/uploads/default/f614d819f8ffe89dc9137735a1936dcbc2cd170f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f614d819f8ffe89dc9137735a1936dcbc2cd170f_2_690x277.png" alt="image" data-base62-sha1="z6W7cjMttcIzmuq6E0qVrCfxfe7" width="690" height="277" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f614d819f8ffe89dc9137735a1936dcbc2cd170f_2_690x277.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f614d819f8ffe89dc9137735a1936dcbc2cd170f_2_1035x415.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f614d819f8ffe89dc9137735a1936dcbc2cd170f_2_1380x554.png 2x" data-dominant-color="F7E1D8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1831×737 46.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Question about DDPG loss.backward()

username: jayjay
cooked: <p>In DDPG implementation, I have a quick question about loss.backward()</p>
<p>Following is the loss.update part of DDPG</p>
<p>target_Q = critic_target(next_state, actor_target(next_state))<br>
target_Q = reward + (done * args.gamma * target_Q).detach()<br>
current_Q = critic(state, action)<br>
critic_loss = F.mse_loss(current_Q, target_Q)</p>
<p>critic_optimizer.zero_grad()<br>
critic_loss.backward()<br>
critic_optimizer.step()</p>
<p>actor_loss = -critic(state, actor(state)).mean()</p>
<p>actor_optimizer.zero_grad()<br>
actor_loss.backward()<br>
actor_optimizer.step()</p>
<p>My question is that once critic_loss.backward() is done, gradients(from critic_loss to each leaf variables end) are computed and when actor_loss.backward() is called, these gradients made in critic_loss.backward() may affect gradients values for actor networks (because of actor_loss form(=critic(state, actor(state)))?</p>

username: vmoens
cooked: <p>Yeah, in some implementations people just don’t care about that because they will call zero grad on the critic just after (like you do).<br>
But in practice it’s best not to backprop through the critic at all to avoid surprises (eg when copy pasting the code).<br>
If someone is calling <code>zero_grad</code> after the optimizer step (which is generally ok) in this case it will be a problem.<br>
Also, you’re spending time computing gradients that you’re not going to use.</p>
<p>TL;DR: best to exclude your critic when you’re comptuting grads for the actor.</p>
Minibatch training with class weight

username: ruccidavinci
cooked: <p>I’m wondering whether we can control class distribution from dataloader to produce every minibatch to have the same class distribution or not.  I understand that we can apply class weight to the loss function, however, it is ok for batch training but I’m not sure about model learning behavior when apply the fixed class weight to minibatch training that each minibatch could have different class distribution.</p>
<p>Or … we apply the fixed class distribution to the loss function and also apply sample weight to multiply directly to the loss for each minibatch?</p>

username: ptrblck
cooked: <p>You could use a <code>WeightedRandomSampler</code> as described <a href="https://discuss.pytorch.org/t/how-to-handle-imbalanced-classes/11264/68">here</a>. Note that it’s a random sampling process and thus a perfectly balanced batch is not guaranteed.</p>

username: ruccidavinci
cooked: <p>Am I understand correctly that it is valid methodology to define just a cross entropy loss (for example) without assigning class weight but calculate the sample weight in each minibatch to multiply directly with the loss?</p>

username: ptrblck
cooked: <p>You are not multiplying the loss. The <code>WeightedRandomSampler</code> uses the passed weights to draw samples from the dataset ideally creating a balanced batch. The loss calculation is then a standard one without any modifications or weighting.</p>
TorchSummary Summary not working on mixed input architectures

username: Ifyanene
cooked: <pre><code class="lang-auto">class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=3),
            nn.ReLU(),
            nn.Conv2d(16, 32, kernel_size=3),
            nn.ReLU(),
            nn.Conv2d(32, 2, kernel_size=3),
            nn.ReLU(),
            nn.MaxPool2d(2, stride=2),
            nn.Flatten(),
            nn.Linear(26, 10)
        )

        self.fcn = nn.Sequential(
            nn.Linear(256, 96),
            nn.ReLU(),
            nn.Linear(96, 48),
            nn.ReLU(),
            nn.Linear(48, 24)
        )

        self.combined = nn.Sequential(
            nn.Linear(34, 20),
            nn.ReLU(),
            nn.Linear(20, 15),
            nn.Sigmoid()
        )



    def forward(self, image1, data):
        x_image = self.encoder(image1)
        x_fcn = self.fcn(data)
        x_multi = torch.cat((x_image , x_fcn), dim=1)
        return self.combined(x_multi)
</code></pre>
<p>I have an architecture that looks like this and I am trying to view the summary (and get out the number of parameters) from this model, however when I run <code>summary(model, [(1, 32, 8), (256,)], device='cpu')</code> I get out a result showing the Layer(type) output shape etc… and just after this when it’s meant to print the number of parameters, it prints <code>can't multiply sequence by non-int of type 'tuple'</code>. I can include the entire stack trace if that helps. I’m not sure how to fix this problem.</p>

username: ptrblck
cooked: <p>Use <code>torchinfo</code> as <code>torchsummary</code> is deprecated and didn’t receive updates in a few years.</p>

username: Ifyanene
cooked: <p>Hi, thanks for your response. I installed torchinfo-1.5.4, I’m using python 3.6, and tried to run <code>summary(model, [(1, 32, 8), (1, 256)], device='cpu')</code> using torchinfo’s summary and I get a runtime error saying <code>Failed to run torchinfo. See above stack traces for more details. Executed layers up to: []</code>. I 'm not sure what to do.</p>
Trying to locate body of function torch.compile()

username: jury_duty
cooked: <p>I see following functions available on torch library:</p>
<p>pwd<br>
/usr/local/lib64/python3.9/site-packages/torch<br>
<strong>init</strong>.py:1610:def compile(model: Optional[Callable] = None, *,<br>
_inductor/<strong>init</strong>.py:8:def compile(<br>
_inductor/codecache.py:936:    def compile(cls, graph, source_code, cuda):<br>
compiler/<strong>init</strong>.py:13:def compile(*args, **kwargs):<br>
distributed/_spmd/api.py:498:def compile(<br>
nn/modules/module.py:2536:    def compile(self, *args, **kwargs):<br>
onnx/_internal/onnxruntime.py:882:    def compile(self, graph_module: torch.fx.GraphModule, args) → torch.fx.GraphModule:</p>
<p>For each one, I instrumented code to insert debug statement to see which one is called by torch.compile() but nothing prints out. Am I missing something? Or is it because it is cached somewhere?</p>
<p>_inductor/<strong>init</strong>.py:13:    print(“GH: compile()-2”)<br>
grep: _inductor/<strong>pycache</strong>/<strong>init</strong>.cpython-39.pyc: binary file matches<br>
compiler/<strong>init</strong>.py:17:    print(“GH: compile()”)<br>
grep: compiler/<strong>pycache</strong>/<strong>init</strong>.cpython-39.pyc: binary file matches<br>
distributed/_spmd/api.py:503:    print(“GH: compile()-3”)</p>

username: bdhirsh
cooked: <p><code>torch.compile</code> won’t actually compile anything until the first time you invoke your compiled function/module. This is because torch.compile will (potentially) specialize on your particular sample inputs to generate efficient code.</p>
<p>If you want to e.g. put a breakpoint in inductor and look at the graph that inductor is compiling, you’d need to actually invoke the compiled function:</p>
<pre><code class="lang-auto">def f(x):
    return x.sin() + x.cos()

x = torch.ones(4)
compiled_f = torch.compile(f)
# run compiled_f, which will actually invoke all of the logic in torch.compile:
out = compiled_f(x)
</code></pre>
Wrong with code_coverage

username: BiophiliaSWDA
cooked: <p>When I run <code>python oss_coverage.py --run-only=test_complex.py</code>, I get a Warning <code>pytorch/tools/code_coverage/package/util/../../profile/raw/test_complex.py.profraw: unsupported instrumentation profile format version</code> and a Error  <code>error: no profile can be merged</code></p>
<p>The results as follows:</p>
<pre><code class="lang-auto">(pytorchcov) myconda@.. code_coverage % python oss_coverage.py --run-only=test_complex.py
....
----------------------------------------------------------------------
Ran 4 tests in 0.002s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.002s

OK
....
----------------------------------------------------------------------
Ran 4 tests in 0.002s

OK
start merge
start merge test_complex.py.profraw
warning: /Users/../pytorch/tools/code_coverage/package/util/../../profile/raw/test_complex.py.profraw: unsupported instrumentation profile format version
error: no profile can be merged
Traceback (most recent call last):
  File "/Users/../pytorch/tools/code_coverage/oss_coverage.py", line 24, in &lt;module&gt;
    report_coverage()
  File "/Users/../pytorch/tools/code_coverage/oss_coverage.py", line 15, in report_coverage
    get_json_report(test_list, options)
  File "/Users/../pytorch/tools/code_coverage/package/oss/cov_json.py", line 17, in get_json_report
    clang_coverage.merge(test_list, TestPlatform.OSS)
  File "/Users/../pytorch/tools/code_coverage/package/tool/clang_coverage.py", line 117, in merge
    merge_target(raw_file, merged_file, platform_type)
  File "/Users/../pytorch/tools/code_coverage/package/tool/clang_coverage.py", line 57, in merge_target
    subprocess.check_call(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/opt/llvm/bin/llvm-profdata', 'merge', '-sparse', '/Users/../pytorch/tools/code_coverage/package/util/../../profile/raw/test_complex.py.profraw', '-o', '/Users/../pytorch/tools/code_coverage/package/util/../../profile/merged/test_complex.py.merged']' returned non-zero exit status 1.
</code></pre>
<p>After reading the Error, I run <code>llvm-profdata merge -sparse /Users/../pytorch/tools/code_coverage/package/util/../../profile/raw/test_complex.py.profraw -o /Users/../pytorch/tools/code_coverage/package/util/../../profile/merged/test_complex.py.merged</code><br>
I get</p>
<pre><code class="lang-auto">warning: /Users/../pytorch/tools/code_coverage/package/util/../../profile/raw/test_complex.py.profraw: unsupported instrumentation profile format version
error: no profile can be merged
</code></pre>
<p>I would like to know what causes this problem and how to fix it.</p>
The meaning of the batch in DataParallel op

username: Danfeng_Zhang
cooked: <p>Hi,<br>
When using torch.nn.DataParallel(), we obtain batch data through dataset  such as  ,for batch in dataset. At this time, should the batch in the dataset be set to the batch of all cards or a single card?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>The deprecated <code>nn.DataParallel</code> module will split the input batch and send each chunk to the corresponding GPU. The batch size is this the “global” batch size.</p>
Custom splitting torchvision datasets

username: Chirath_Pansilu
cooked: <p>I want to use CIFAR100 dataset from the <code>torchvision.datasets</code>. Also I want to do a custom split for train, test and validation dataset. Since torchvision only provides train and test datasets I was going to concatenate the train and test datasets. And use a <code>custom_split</code> for train, test and validation testsets. Is there an easier way to do this or is this the procedure I should follow?</p>

username: ptrblck
cooked: <p>Your approach sounds valid and I assume you would use <code>torch.utils.data.Subset</code> and pass the corresponding indices to the splits?</p>

username: Chirath_Pansilu
cooked: <p>I was going to use <code>custom_split</code> with fractions. 60% for training set and 20% each for test and validation sets. Also is it valid to add <code>transforms</code> to each train and test dataset given by <code>torchvision.datasets</code> and then concatenate them later? Are those transformations also applied to the concatenated dataset as well?</p>
Gradient Calculation for part of the mini-batch

username: J_B_28
cooked: <p>Hi, I am currently considering a special case of training the model, which I only use part of the data to compute the loss, for example:</p>
<pre><code class="lang-auto">def train(epoch):
    model.train()
    total_loss = total_correct = 0
    for batch in train_loader:
        optimizer.zero_grad()
        out = model(batch.x, batch.edge_index.to(device))[:100]
        y = batch.y[:100].squeeze()
        loss = F.cross_entropy(out, y)
        loss.backward()
        optimizer.step()

        total_loss += float(loss)
        total_correct += int(out.argmax(dim=-1).eq(y).sum())
</code></pre>
<p>I have two questions about the backward propagation:</p>
<p>(1) What are the gradients for those data points that do not participate in the loss computation? For instance, in the given example, what are the gradients for data points beyond the first 100? Are they zero?</p>
<p>(2) In this case, which part of the data will participant in update the model? Do those data points that are not part of the loss computation (say after first 100 in the above example) contribute to the update of the model parameters?</p>
<p>Thanks.</p>

username: soulitzer
cooked: <ol start="2">
<li>
<p>If you have any operations such as batch normalization that operate on the entire batch of data, then data points beyond the first 100 can also contribute to the update of parameters.</p>
</li>
<li>
<p>If your answer to (2) was yes, then gradients may be non-zero; otherwise they would be zero yes. Note that typically data points shouldn’t require gradients, only parameters should have gradients.</p>
</li>
</ol>

username: J_B_28
cooked: <p>Thanks for your reply! One following question: in this scenario, is it equivalent to using two input tensors, <code>x1</code> with <code>requires_grad = True</code> of length 100 and <code>x2</code> with <code>requires_grad = False</code> of length (original batch size - 100), then concatenating them together as the input? Thanks again!</p>
RuntimeError: Function ‘UnsafeViewBackward0’ returned nan values in its 0th output

username: migellebarlis
cooked: <p>Hi,</p>
<p>I wanted to trace why the losses, and psnr values of my model is NaN. Some of the articles here pointed out to use with torch.autograd.detect_anomaly(True): After doing this, an error is thrown “RuntimeError: Function ‘UnsafeViewBackward0’ returned nan values in its 0th output.” I’m not sure where to start since I do not know what is UnsafeViewBackward0. And how should I trace where the values became NaN.</p>
<p>Thanks!</p>

username: soulitzer
cooked: <p>_unsafe_view is a lower-level operation that can called by other ops. One way forward is to try logging out the ops to get sense of what the other ops around it are:</p>
<pre><code class="lang-auto">from torch.testing._internal.logging_tensor import capture_logs_with_logging_tensor_mode
x = torch.randn([])
y = torch.randn([])
with capture_logs_with_logging_tensor_mode() as logs:
    torch.empty([])
    x + y
print('\n'.join(logs))
</code></pre>
<pre><code class="lang-auto">$0 = torch._ops.aten.empty.memory_format([], device=device(type='cpu'), pin_memory=False)
$3 = torch._ops.aten.add.Tensor($1, $2)
</code></pre>
Multi-Input data of different sizes

username: Ifyanene
cooked: <p>I am working on a multi (or mixed) input problem, whereby each row in my tabular dataset has two accompanying images, hence if I have a tabular data with 300 rows, I will have 600 images.</p>
<p>I created a custom dataset class for this problem, however because I have 600 images and 300 rows, my len method will return only 300 which means I’m basically discarding half my images. I’m wondering if I have the right idea or if someone can point me in the right direction. I was initially thinking of adding an extra argument to the dataset’s constructor (i.e image1 and image2), but I think this means I will have to create two CNN’s (one for each image) and I think this might be overkill for my problem.</p>