akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
SWA for distributed training

Smith_John1: <p>Hi there,</p>
<p>I am very new to pytorch. Is Stochastic Weight Averaging supported in distributed training (more specifically, the update_bn function).</p>
<p>Thanks!</p>

cbalioglu: <p>I see no reason why it wouldn’t work. Since SWA holds a separate copy of the model on each rank and uses the same all-reduced gradients to update the weight averages, it should theoretically work.</p>
<p>I would suggest giving it a try, and letting us know if you experience any issues.</p>

nsriniva03: <p>I have tried SWA with DDP and encounter key errors when I try to load the state_dict of the saved swa model.</p>
<p>Below is my skeletal code:</p>
<pre><code class="lang-auto">
from torch.optim.swa_utils import AveragedModel, SWALR
from torch.optim.lr_scheduler import CosineAnnealingLR

loader, optimizer, model, loss_fn = ...
model = torch.nn.parallel.DistributedDataParallel(module=model, broadcast_buffers=False,  device_ids=[local_rank])
model.train()
swa_model = AveragedModel(model)
scheduler = CosineAnnealingLR(optimizer, T_max=100)
swa_start = 5
swa_scheduler = SWALR(optimizer, swa_lr=0.05)

for epoch in range(100):
      for input, target in loader:
          optimizer.zero_grad()
          loss_fn(model(input), target).backward()
          optimizer.step()
      if epoch &gt; swa_start:
          swa_model.update_parameters(model)
          swa_scheduler.step()
      else:
          scheduler.step()

# Update bn statistics for the swa_model at the end
torch.optim.swa_utils.update_bn(loader, swa_model)
#saving swa_model
torch.save(swa_model.module.state_dict(),  'swa_model.pth'))
#saving model at last epoch
torch.save(model.module.state_dict(),  'last_epoch_model.pth'))
# The code runs fine and the model is saved
#During inference

model = get_model()
model.load_state_dict(torch.load('swa_model.pth'))

</code></pre>
<p>Now there is a mis-match of keys in the state dictionary<br>
Because I have used DDP and passed the DDP model to AverageModels - Results in the following error:<br>
Missing key(s) in state_dict: “conv1.weight”…; Unexpected key(s) in state_dict: “module.conv1.weight”…</p>
<p>But “last_epoch_model.pth”, loads correctly. Any thoughts?</p>

Fredrik_Opeide: <p>When wrapping your model with both DDP and AverageModel, you need to call swa_model.module.module to get back to the correct level. Right now, with only one “.module”, you’re saving a DDP wrapped model, which is the reason for the unexpected ‘module’-prepend in your state_dict keys when loading.</p>
Undefined symbol: When I try to check if my custom operator is registered or not

atakirk: <p>I have implemented custom conv2d function in c++ and I want to test it with pytorch model. I registered my custom operator but when I try to check if my operator is registered correctly or not, I am getting this error: undefined symbol: _ZN8pybind116detail11type_casterIN2at6TensorEvE4loadENS_6handleEb</p>
<p>I am using this python code in order to check if my operator is registered correctly:</p>
<pre><code class="lang-auto">import torch
import sys

def get_registered_operators():
    operators = []
    for op_name in dir(torch.ops):
        if op_name.startswith("__") and op_name.endswith("__"):
            continue
        operators.append(op_name)
    return operators

def is_operator_registered(operator_name):
    return hasattr(torch.ops, operator_name)

if __name__ == "__main__":
    # Load the custom operator library
    shared_library_path = '/path/to/libcustom_direct_conv2d_op.so'
    try:
        torch.ops.load_library(shared_library_path)
    except OSError as e:
        print(f"Error loading library: {e}")
        sys.exit(1)

    # Check if your custom operator is registered
    custom_op_name = "bsc_direct_conv2d"
    if is_operator_registered(custom_op_name):
        print(f"{custom_op_name} is registered.")
    else:
        print(f"{custom_op_name} is NOT registered.")

    # Get all the registered operators
    registered_operators = get_registered_operators()
    print("\nRegistered operators:")
    for operator_name in registered_operators:
        print(operator_name)

</code></pre>
<p>When I run this I got this error:<br>
Undefined symbol: _ZN8pybind116detail11type_casterIN2at6TensorEvE4loadENS_6handleEb<br>
If you can help me about this I would be glad. Thanks in advance!</p>

skluge: <p>I have the exact same Problem with the same undefined symbol. Have you solved this already?</p>
Understanding nn.MultiheadAttention

enerd: <p>I tried to understand the multihead attention implementation, and tried the following:</p>
<pre><code class="lang-auto">embed_dim, num_heads  = 8, 2
mha = nn.MultiheadAttention(embed_dim=embed_dim, num_heads=num_heads, dropout=0, bias=False, add_bias_kv=False, add_zero_attn=False)

seq_len = 2
x = torch.rand(seq_len, embed_dim)

# Self-attention: Reference calculations
attn_output, attn_output_weights=mha(x, x, x)

# My manual calculations
wq, wk, wv = torch.split(mha.in_proj_weight, [embed_dim, embed_dim, embed_dim], dim=0)

q = torch.matmul(x, wq)
k = torch.matmul(x, wk)
v = torch.matmul(x, wv)

dk = embed_dim // num_heads
attention_map_manual = torch.matmul(q, k.transpose(0, 1)) / (math.sqrt(dk))
attention_map_manual = attention_map_manual.softmax(dim=1)

torch.allclose(attention_map_manual, attn_output_weights, atol=1e-4) # -&gt; returns false
</code></pre>
<p>Why it returns zero? What is wrong with my calculations?</p>

enerd: <p>Ok, I figured it out by looking at the source code. To anyone who wants to understand the weights and calculations in the multi-head attention, here is a simple <a href="https://gist.github.com/eneserdo/77b468f61fa5c3c9f4587b4a51fca963" rel="noopener nofollow ugc">gist</a></p>
From distributed to gradient accumulation

External_happy: <p>This code was run perfect on multiple gpu, however, I want to adapted on a single gpu by using gradient accumulation. How to do that ?</p>
<pre><code class="lang-auto">class GatherLayer(torch.autograd.Function):
    """
    Gather tensors from all workers with support for backward propagation:
    This implementation does not cut the gradients as torch.distributed.all_gather does.
    """
    @staticmethod
    def forward(ctx, x):
        output = [torch.zeros_like(x) for _ in range(torch.distributed.get_world_size())]
        torch.distributed.all_gather(output, x)
        return tuple(output)
    @staticmethod
    def backward(ctx, *grads):
        all_gradients = torch.stack(grads)
        torch.distributed.all_reduce(all_gradients)
        return all_gradients[torch.distributed.get_rank()]
def all_gather_with_grad(tensors):
    """
    Performs all_gather operation on the provided tensors.
    Graph remains connected for backward grad computation.
    """
    # Queue the gathered tensors
    world_size = torch.distributed.get_world_size()
    # There is no need for reduction in the single-proc case
    if world_size == 1:
        return tensors

    tensor_all = GatherLayer.apply(tensors)
    return torch.cat(tensor_all, dim=0)


def train(….):
    with amp_autocast():    
            logits = model(images)
            probs = F.softmax(logits,dim=-1) 
            probs_all = utils.all_gather_with_grad(probs)
            probs_batch_avg = probs_all.mean(0) # average prediction probability across all gpus
            probs_avg = probs_batch_avg
            loss_xx = -(torch.log(probs_avg)).mean()
            loss = loss_xx + .......

loss.backward()

</code></pre>
PyTorch 2D Object Detection Training Code Resources

HassanBinHaroon: <p>Could anyone please provide guidance on PyTorch-based 2D Object Detection training code resources? While resources for image classification are available, offering well-known architectures and built-in datasets, I am specifically seeking assistance with 2D Object Detection. Your help in locating relevant materials would be greatly appreciated. Thank you.</p>

ptrblck: <p>Maybe <a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">this tutorial</a> might be useful. I would also not recommend tagging specific users as it might demotivate others to post a valid and potentially better answer.</p>

J_Johnson: <p>If you’re looking for pre-trained models in 2D object detection, you can find those here:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/hub/ultralytics_yolov5/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/hub/ultralytics_yolov5/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/362;"><img src="https://pytorch.org/assets/images/social-share.jpg" class="thumbnail" width="690" height="362"></div>

<h3><a href="https://pytorch.org/hub/ultralytics_yolov5/" target="_blank" rel="noopener nofollow ugc">YOLOv5</a></h3>

  <p>Before You Start</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Profiling Granularity Using torch profiler

karki: <p>Hi everyone!</p>
<p>I’m trying to profile different sections of my code with the torch profiler. But while I want this functionality, I also want to avoid a lot of if-else switches. I did find a way around it by actually splitting the runner loop into a runner loop that only runs for profiling and another one which runs during training. But now I wish to create a more fine grained profiling and want to document different sections as named sections. For example, just the backbone implementation separately. But there are two problems at hand: One is that the section lies outside the runner file so I need a way to pass the profiler object to the forward/backward section and another issue is that I run into a situation with too many indents and if statements within my code which render the code sections almost unreadable or too ugly. Please let me know if you guys have any suggestions or ways to actually work around this.</p>
<p>Please ask me any questions if it helps. Thank you!</p>
Converting tensor into text then performing replace

peony: <p>Hello everyone, I have the following function:</p>
<pre><code class="lang-auto">def get_videoname(paths):

    
    vidname = str(paths)
    vidname = vidname.strip()
    vidname = vidname.replace('[', '')
    vidname = vidname.replace(']', '')
    vidname = vidname.replace(' ', '')
    vidname = vidname.replace(r"D:\\A\\C\\frames-KP\\dev\\", '')
 
    vidname = vidname.split(',')
    return vidname
</code></pre>
<p>and it outputs the following after I loop through the list:</p>
<pre><code class="lang-auto">'K000000_P0000_T00\\000024.jpg.npy'
'K000000_P0000_T00\\000025.jpg.npy'
'K000000_P0000_T00\\000026.jpg.npy'
'K000000_P0000_T00\\000027.jpg.npy'
'K000000_P0000_T00\\000028.jpg.npy'
'K000000_P0000_T00\\000029.jpg.npy'
'K000000_P0000_T00\\000030.jpg.npy'
'K000000_P0000_T00\\000031.jpg.npy'

</code></pre>
<p>How can I edit my function so I can remove *.jpg.npy</p>
<p>I tried <code>re.sub(r'\(*.jpg.npy\)', '', vidname)</code>, but it doesn’t seem to work.</p>

ptrblck: <p>Wouldn’t another <code>replace</code> call work?<br>
Also, how is this related to tensors?</p>
Geforce RTX 4070 Ti Super support for pytorch-gpu

hoya9802: <p>Hi,<br>
Could you please let me know whether “Geforce RTX 4070ti Super” support Pytorch-gpu.<br>
In below link <a href="https://developer.nvidia.com/cuda-gpus" rel="noopener nofollow ugc">https://developer.nvidia.com/cuda-gpus </a><br>
it is specified that Geforce RTX 4070 ti is capable for CUDA Pytorch-gpu. But it didn’t mentioned anything about “4070ti Super”. “RTX 4070 Ti” and “RTX 4070 Ti Super” have different numbers of CUDA cores and VRAM, so I’m concerned about potential issues when configuring environments for Pytorch-GPU in the future. Please confirm whether Geforce RTX 4070ti Super support Pytorch-gpu.</p>

ptrblck: <p>Yes, the 4070 To Super is supported in all of our current binaries.</p>

hoya9802: <p>Thank you so much! Even though they both have different CUDA cores and VRAM, the RTX 4070ti Super can run all of the current binaries using the GPU!</p>

ptrblck: <p>Yes as these details are irrelevant.</p>

hoya9802: <p>Thanks for letting me know=)</p>
Casting Inputs Using custom_fwd Disables Gradient Tracking

BobMcDear: <p>Hello,</p>
<p>I would like to implement automatic mixed precision for some custom autograd functions, but decorating them using <code>custom_fwd(cast_inputs=torch.float16)</code> disables gradient tracking for the inputs unless they are already of the desired type (that is, when casting does not occur), as demonstrated below.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch # Version 2.2.0
from torch.cuda.amp import autocast, custom_fwd


class IdentityFunc(torch.autograd.Function):
    @staticmethod
    @custom_fwd(cast_inputs=torch.float16)
    def forward(ctx, input):
        print(input.dtype, input.requires_grad)
        return input


input = torch.tensor(0, dtype=torch.float32, device='cuda', requires_grad=True)
with autocast():
    output = IdentityFunc.apply(input) # Prints torch.float16 False

input = torch.tensor(0, dtype=torch.float16, device='cuda', requires_grad=True)
with autocast():
    output = IdentityFunc.apply(input) # Prints torch.float16 True
</code></pre>
<p>This idiosyncrasy is not mentioned in the docs, so it might be a bug? If so, I can file an issue on GitHub. Otherwise, is there a means of ensuring tensors retain their <code>requires_grad</code> values through the casting process? In many cases, it is necessary to infer whether a particular input requires its gradients to be calculated during the backward pass, which cannot be done due to this behaviour.</p>
<p>Thank you.</p>

ptrblck: <p>The reason for the difference is due to a cast and the disabled autograd inside the <code>forward</code> and <code>backward</code> methods unrelated to amp:</p>
<pre data-code-wrap="python"><code class="lang-python">class IdentityFunc(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        print(input.dtype, input.requires_grad)
        a = input * 2
        print(a.requires_grad)
        return input * 2
    
    @staticmethod
    def backward(ctx, grad):
        return grad * 2

input = torch.tensor(0, dtype=torch.float32, device='cuda', requires_grad=True)
output = IdentityFunc.apply(input) 
# torch.float32 True
# False
</code></pre>
<p>Since the first code will apply an operation (the <code>to(torch.float16)</code> cast) in the <code>input</code>, the new <code>input</code> tensor will set its <code>.requires_grad</code> attribute to <code>False</code> in the same way it happend for <code>a</code> in my example.<br>
I don’t think there is an easy solution using the <code>custom_fwd</code> decorator while still keeping the <code>.requires_grad</code> attribute alive and you might need to cast the inputs manually in your custom <code>autograd.Function</code> if its called inside an <code>autocast</code> region.</p>

BobMcDear: <p>Thank you for the explanation, that makes sense. I will either manually cast the inputs as you suggest or pass arguments to the forward pass of my custom function denoting which tensors need their gradients computed.</p>
Training on Multiple GPU with Transformers Library

sanipanwala: <p>Hello,</p>
<p>I have two GPUs and during training, I’m getting below exception.</p>
<pre><code class="lang-auto">/cuda/IndexKernel.cu:92: operator(): block: [98,0,0], thread: [64,0,0] Assertion `-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/cuda/IndexKernel.cu:92: operator(): block: [98,0,0], thread: [65,0,0] Assertion `-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/cuda/IndexKernel.cu:92: operator(): block: [98,0,0], thread: [66,0,0] Assertion `-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/cuda/IndexKernel.cu:92: operator(): block: [98,0,0], thread: [67,0,0] Assertion `-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/cuda/IndexKernel.cu:92: operator(): block: [98,0,0], thread: [68,0,0] Assertion `-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/cuda/IndexKernel.cu:92: operator(): block: [98,0,0], thread: [69,0,0] Assertion `-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/cuda/IndexKernel.cu:92: operator(): block: [98,0,0], thread: [70,0,0] Assertion `-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/cuda/IndexKernel.cu:92: operator(): block: [98,0,0], thread: [71,0,0] Assertion `-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.
/cuda/IndexKernel.cu:92: operator(): block: [98,0,0], thread: [72,0,0] Assertion `-sizes[i] &lt;= index &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"` failed.

trainer.train()
  File "/python3.12/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
</code></pre>
<p>I’m using Trainer from Transformers Library.</p>
<pre><code class="lang-auto">model.is_parallelizable = True
model.model_parallel = True
</code></pre>
<p>Can someone please help me with this?</p>
<p>Thanks.</p>

ptrblck: <p>The error points to an indexing error which is most likely caused by an embedding receiving an input containing out-of-bounds values.</p>

sanipanwala: <p>Thanks for the reply <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>How can I track and resolve this issue/</p>
<p>Thanks.</p>

ptrblck: <p>Rerun your code with blocking launches via <code>CUDA_LAUNCH_BLOCKING=1</code> as described in your error message (which is missing here) and isolate the failing operation. Once done, check the inputs and the min/max values and make sure they are valid.</p>
Data parallelism, gradient accumulation and batch size

lkm9983: <p>Hello,<br>
I’m trying to employ larger batch sizes.<br>
There are several techniques to implement large batch sizes:<br>
data parallelism, gradient accumulation and batch size</p>
<p>I did toy examples and I found they are not the exact same results.</p>
<p>I did the three settings:</p>
<ol>
<li>batch size 64 with 1 gpu</li>
<li>batch size 32 with 2 GPUs</li>
<li>batch size 32 with 1 gpu and gradient accumulation steps 2</li>
</ol>
<p>I used BertModels and mrpc dataset from Glue and Huggingface Trainer.(I also set seed)</p>
<p>Someone said that scaling learning rate is necessary.<br>
<a href="https://github.com/Lightning-AI/pytorch-lightning/discussions/3706" rel="noopener nofollow ugc">How to scale learning rate with batch size for DDP training? #3706</a><br>
But I don’t think that the gradient from 2 GPUs is not the random variables. I think they are deterministic when the seed is set.</p>
<p>Why the results are different?</p>

ptrblck: <aside class="quote no-group" data-username="lkm9983" data-post="1" data-topic="196798">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lkm9983/48/58653_2.png" class="avatar"> lkm9983:</div>
<blockquote>
<p>I think they are deterministic when the seed is set.</p>
</blockquote>
</aside>
<p>That’s not the case and you need to enable deterministic algorithms as explained <a href="https://pytorch.org/docs/stable/notes/randomness.html">in the docs</a>. However, even then you cannot expect to receive bitwise-identical results since the workloads differ by batch size, different order of communication etc.</p>
Using multiple outputs as inputs of a second part network before backward

Bruno2: <p>Hello,<br>
I have been trying to train the model below:</p>
<pre><code class="lang-auto">class CombinedModels(nn.Module):
    def __init__(self):
        super().__init__()
        self.channels = 32
        self.feature_extractor = Densenet_Feature()
        self.Regressor = Fully_Connected_Regressor(self.channels)

    def forward(self, x): # X would be of shape [batch size, same as self.channels, width, height]
        features = torch.zeros((x.shape[0], self.channels*100), device="cuda", dtype=torch.float16)
        for i in range(self.channels):
            features[:,i*100:(i+1)*100] = self.feature_extractor(x[:, i:i+1, :, :])
        return self.Regressor(features)

def Densenet_Feature():
    model = models.densenet161(weights= models.DenseNet161_Weights.DEFAULT)
    starting_weights = model.features[0].weight
    new_conv_layer = nn.Conv2d(1, 96, kernel_size=7, stride=2, padding=3, bias=False)
    new_conv_layer.weight.data[:,0,:,:] = (starting_weights[:,0,:,:]+ starting_weights[:,1,:,:] + starting_weights[:,2,:,:])/3

    model.features[0] = new_conv_layer

    # Compress to 100 features
    model.classifier = nn.Linear(2208, 100)

    return model

def Fully_Connected_Regressor(channels):
    model = nn.Sequential(nn.Linear(100*channels, 100), nn.Linear(100, 8))
    return model


</code></pre>
<p>The code runs fine but it doesn’t seem as the gradient is updating correctly.<br>
I tested it passing the exact same data as training and testing and the model doesn’t seem to converge, while passing the same data in a more traditional way (as a 32 channel image) converges nicely.<br>
I am assuming there is some issue with the concatenation of the outputs of the first Densenet before going over the classifier that breaks the backward pass in some way?</p>
<p>Thanks!</p>
Detect Entity for Semantic Parsing with Generative Model

jupyter: <p>Hi,</p>
<p>I’m doing Semantic Parsing problem for Question Answering Knowledge Graph.<br>
Here is an example:<br>
Q: Who is Barack Obama?<br>
Cypher: MATCH (n) WHERE n.name = “Barack Obama”</p>
<p>I used RNN-based Seq2Seq architecture as my baseline to check if the model is capable of capturing new entities appear in the question. As my corpus is limited, there are a lot of unseen entities (unseen token). I have try to modify the output which is just Barack Obama instead of the whole query sentence. But it seems doesn’t work with unseen entities. Like the model cannot copy paste the entities string in the input sentence into the output sentence. Should I train another NER model or using a pretrained token embedding will help in this case?</p>
Pytorch multiprocessing

Reha_Mathur1: <p>Hi! I am trying to use pytorch to solve an optimization problem with gradient descent. Basically, I have a model with a parameter v and over each of my 7 experiments, the model sequentially runs a forward process and calls the calculate_labeling function with v as the input. The output of that forward process is aggregated and then sent to the loss function which is then backpropagated to update v. I want to parallelize running the forward process on each of the experiments with pytorch.multiprocessing and I have access to a multicore cpu.</p>
<p>This is the general code i have been using but i ran into this error. Does anyone have any tips on how I could either speed up this process in a different way or achieve this parallelization? The backwards calls on each of these forward calls really take up the bulk of the time so i was hoping to parallelize those specifically.</p>
<p><code>RuntimeError: Cowardly refusing to serialize non-leaf tensor which requires_grad, since autograd does not support crossing process boundaries.  If you just want to transfer the data, call detach() on the tensor before serializing (e.g., putting it on the queue).</code></p>
<pre><code class="lang-auto">def forward(self):
        output_queue = mp.Queue()
        
        # Container for processes
        processes = []

        # Loop over experiments and spawn a process for each calculation
        for the_experiment in self.experiments:
            p = mp.Process(target=self.worker, args=(the_experiment, v, self.device, output_queue))
            p.start()
            processes.append(p)

        # Collect results
        labeling_results = [output_queue.get() for _ in self.experiments]

        # Make sure all processes have finished
        for p in processes:
            p.join()

        return torch.stack(labeling_results).squeeze(0)

    @staticmethod
    def worker(the_experiment, v, device, output_queue):
        the_labeling = calculate_labeling(v, device)
        
        output_queue.put(the_labeling)
</code></pre>
Torchvision v2.Grayscale not returning 1 channel

musl: <p>Hi,<br>
It looks like torchvision v2.Grayscale is not returning 1 channel, even if passing <code>num_output_channels=1</code>:</p>
<pre><code class="lang-auto">import cv2
from torchvision.transforms import v2

img = cv2.imread('/tmp/test.png')
gray = v2.Grayscale(num_output_channels=1)
out_img = gray(img)
print(f'{out_img.shape}')
# (339, 510, 3)
</code></pre>
<p>Does anybody knows a solution? Using <code>functional.rgb_to_grayscale</code> works as intended (outputs only 1 channel), however that function doesn’t work with numpy arrays, only PIL or tensors. Which means I need to do a whole lot of conversions for it to work:</p>
<pre><code class="lang-auto">import numpy as np
from PIL import Image
from torchvision.transforms.functional import rgb_to_grayscale

# my inputs are float32, but PIL can't read it
img = Image.fromarray(img_gt.astype(np.uint8))
img = rgb_to_grayscale(img)
img = np.array(img, dtype=np.float32)
# my other functions expect it to output (h, w, 1), but rgb_to_grayscale outputs (h, w) only
img = np.expand_dims(img, axis=-1)
</code></pre>
<p>Any help is appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Indeed it works with <code>PIL.Image</code>s but seems to fail with <code>numpy.array</code>s (loaded via OpenCV).<br>
CC <a class="mention" href="/u/pmeier">@pmeier</a></p>
BCE with pos_weight will never have 0 gradient?

daturkel: <p>I want to use BCEWithLogitsLoss with a <code>pos_weight</code> value &gt; 1 to deal with class imbalance in my dataset. However, one thing I noticed was that the <code>pos_weight</code> aspect makes the loss always have a nonzero gradient.</p>
<p>L = - pos_weight * y * log(yhat) - (1-y) * log(1 - yhat)<br>
dL/dyhat = -pos_weight * y / yhat + (1 - y) / (1 - yhat)</p>
<p>Even if my model perfectly predicts yhat=y, then my gradient is -pos_weight yhat/yhat + (1-yhat) / (1-yhat) = -pos_weight + 1 &lt; 0 for pos_weight &gt; 0.</p>
<p>This seems odd, because it suggests that the model will continue to make (potentially large) updates even if we’re already making correct predictions. Am I misunderstanding something here?</p>
VAE framework question

sswwhh: <p>Hi, Deep learning people</p>
<p>I am new to VAE and wonder if the following framework fits with VAE in pytorch. As I am working on bioinformatics, let me explain the high-level idea based on biology.</p>
<p>Let’s say I have 10 mRNA isoforms (or you can just call it isoforms). These 10 isoforms came from the same gene as one gene can contain multiple isoforms due to alternative splicing. Anyway, I want to see which of these 10 isoforms is the most probable based on the 10 features that I have per isoform. For now let’s just say there are 10 features, so every isoform has 10 features. At the moment, I don’t know which one of those 10 features is important. Is it possible to train those 10 features of each isoform (so in this case there will be 10 * 10 = 100) and find which isoform is the most probable based on those 10 features? If this is possible, can you rank those 10 isoforms based on the latent space or reconstruction error in the decoder that learned the distribution of 10 features?</p>
<p>Or instead, is it common to just use the latent space of the VAE and feed that as an input to a feed-forward neural network to rank those 10 isoforms? In this case, let’s assume we have true labels.</p>
RNN time series predict y outputs after time t with no x input

rcalix: <p>Hello,<br>
I am trying to develop a time series RNN. I use a time window of 8 vectors (t-7 to t) and my goal is to predict a sequence (t+1 to t+8) of regression values y for some chemical. The inputs x at every time step are vectors of size 6 of other chemicals, temp, pressures, etc. The inputs x and outputs y are not the same. I would have these x vectors of size 6 from t-7 to t. Since I am predicting a sequence of 8 values into the future, I will not know the values of the x vectors of size 6 after time “t”. What input do I use for the RNN from time t+1 to t+8? Is there an option where the RNN simply does not have inputs after a certain point in time? I understand that I could simply give it input vectors of size 6 with random data in them or just zeros after time t. But is this a valid approach? I have been searching on line and have not been able to answer this question.<br>
Thank you.</p>

rcalix: <p>Hello,<br>
Any help with this question?<br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Pruning nn.Linear weights causing unexpected errors

arrmansa: <h1><a name="this-fails-1" class="anchor" href="#this-fails-1"></a>This fails</h1>
<pre data-code-wrap="python"><code class="lang-python">import torch

def test1():  
  layer = nn.Linear(100, 10)
  x = 5 - torch.sum(layer(torch.ones(100)))
  x.backward()
  layer.weight.data = layer.weight.data[:, :90]
  layer.weight.grad.data = layer.weight.grad.data[:, :90]
  x = 5 - torch.sum(layer(torch.ones(90)))
  x.backward()
test1()
</code></pre>
<p>with error</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-3-bb36a010bd86&gt; in &lt;cell line: 10&gt;()
      8     x = 5 - torch.sum(layer(torch.ones(90)))
      9     x.backward()
---&gt; 10 test1()
     11 # and this works as well
     12 

2 frames
/usr/local/lib/python3.10/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    249     # some Python versions print out the first line of a multi-line function
    250     # calls in the traceback and some print out the last line
--&gt; 251     Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    252         tensors,
    253         grad_tensors_,

RuntimeError: Function TBackward0 returned an invalid gradient at index 0 - got [10, 90] but expected shape compatible with [10, 100]
</code></pre>
<h1><a name="this-works-2" class="anchor" href="#this-works-2"></a>This works</h1>
<pre data-code-wrap="python"><code class="lang-python">import torch

def test2():  
  layer = torch.nn.Linear(100, 10)
  x = 5 - torch.sum(layer(torch.ones(100)))
  x.backward()
  del x    #main change
  layer.weight.data = layer.weight.data[:, :90]
  layer.weight.grad.data = layer.weight.grad.data[:, :90]
  x = 5 - torch.sum(layer(torch.ones(90)))
  x.backward()
test2()
</code></pre>
<h1><a name="and-this-works-as-well-3" class="anchor" href="#and-this-works-as-well-3"></a>and this works as well</h1>
<pre data-code-wrap="python"><code class="lang-python">import torch
def test3():  
  layer = torch.nn.Linear(100, 10)
  x = 5 - torch.sum(layer(torch.ones(100)))
  x.backward()
  layer.weight.data = layer.weight.data[:, :90]
  layer.weight.grad.data = layer.weight.grad.data[:, :90]
  layer.weight = torch.nn.Parameter(layer.weight)   #main change
  x = 5 - torch.sum(layer(torch.ones(90)))
  x.backward()
test3()
</code></pre>
<p>I encountered this when trying to implement a paper on model pruning. I believe this has something to do with the autograd graph, but I have am not sure what exactly is going on. Any explanation these almost identical code snippets work/fail would be very appreciated. Thanks in advance.</p>
Torch.jit.save not working on some models from Torchhub

LewsTherin: <p>Hi! I’m having issues with exporting some models from Torchhub to Torchscript.</p>
<p>For example, if I try:</p>
<pre><code class="lang-auto">model_name = "x3d_xs"  # this does NOT work
model_name = "i3d_r50"  # this works
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = torch.hub.load("facebookresearch/pytorchvideo", model_name, pretrained=True).to(device)

# train your model here...

torch.jit.save(torch.jit.script(model), "some_saved_model.pt")
</code></pre>
<p>this works totally fine for models such as “i3d_r50”, but if I try models like “x3d_s” or “x3d_xs”, I get the error:</p>
<pre><code class="lang-auto">File "[...]/lib/python3.10/site-packages/torch/jit/annotations.py", line 513, in ann_to_type
    raise ValueError(f"Unknown type annotation: '{ann}' at {loc.highlight()}")
ValueError: Unknown type annotation: '&lt;class 'torch.nn.modules.module.Module'&gt;' at 
python-BaseException
</code></pre>
<p>I did have this issue first when replacing the last layer of the model to classify on custom classes, but the behaviour is the same even if I leave the model as it is.<br>
Am I doing something wrong here?</p>
Torch.split without explicitly indicating split_size_or_sections

peony: <p>If I have the following tensor:</p>
<pre><code class="lang-auto">tensor([[0, 1],
        [2, 3],
        [4, 5],
        [6, 7],
        [8, 9]])
</code></pre>
<p>I know I can split it into 5 chunks using <code>torch.split(t, 5)</code> . However, I’m interested in automatically determining the number of splits based on the number of brackets enclosing the data. In this case, there’s one main bracket encapsulating 5 inner brackets, so I want to split it into 5 chunks automatically without specifying <code>split_size_or_sections</code> . How can I achieve this in PyTorch?</p>
<p>Thank you in advance</p>

ptrblck: <aside class="quote no-group" data-username="peony" data-post="1" data-topic="196765">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e5b9ba/48.png" class="avatar"> peony:</div>
<blockquote>
<p>However, I’m interested in automatically determining the number of splits based on the number of brackets enclosing the data.</p>
</blockquote>
</aside>
<p>I assume you want to split in <code>dim0</code> into single slices?<br>
If so,</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([[0, 1],
                  [2, 3],
                  [4, 5],
                  [6, 7],
                  [8, 9]])

x.split(split_size=1)
</code></pre>
<p>will work as it will return 5 slices.</p>

peony: <p>Oh so I figured it out, I can simply use <code>t.size(dim=1)</code> as the <code>split_size_or_sections</code>. But it doesn’t split the tensor, here is my code:</p>
<pre><code class="lang-auto">centlen = centroids.size(dim=1)
splitcentroids = torch.tensor_split(centroids, centlen, dim=0)

for x in splitcentroids:
         print('x', x)
</code></pre>
<p>and the output is something like this:</p>
<pre><code class="lang-auto">x tensor([[[ 1,2],
         [ 3,4],
         [ 5,6]]], device='cuda:0')
x tensor([], device='cuda:0', size=(0, 3, 2))
x tensor([], device='cuda:0', size=(0, 3, 2))
</code></pre>
<p>However, what I want is something like:</p>
<pre><code class="lang-auto">
x tensor([1,2])
x tensor([3, 4])
x tensor ([5, 6])
</code></pre>
<p>Any advice for me? Thank you in advance</p>

peony: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I tried it but when I did a loop like this</p>
<pre><code class="lang-auto">for x in splitcentroids:
         print('x', x)
</code></pre>
<p>it outputs the following</p>
<pre><code class="lang-auto">tensor([[[0, 1],
       [2, 3],
       [4, 5],
       [6, 7],
       [8, 9]]])
</code></pre>
<p>instead of:</p>
<pre><code class="lang-auto">x tensor([1,2])
x tensor([3, 4])
x tensor ([5, 6])
</code></pre>

peony: <p>Oh I realized I was using the wrong dim. The solution worked, thanks for the help…</p>
Where do I find the fundamental CUDA kernels underlying PyTorch?

Max_Unhold: <p>Hello,<br>
Pytorch has very optimized matrix operations when it comes to the standart operations<br>
such as torch.matmul, torch.sparse.mm, …</p>
<p>But where do I find the most fundamental CUDA kernels for these algorithms when I want to participate in the development?</p>
<p>I have been looking in the <strong>PyTorch</strong> packages as well as in the <strong>ATen</strong> library but was not able to find them in e.g. the form</p>
<pre data-code-wrap="cuda"><code class="lang-cuda">// CUDA Kernel function to add the elements of two arrays on the GPU
__global__
void add(int n, float *x, float *y)
{
  for (int i = 0; i &lt; n; i++)
      y[i] = x[i] + y[i];
}
</code></pre>
<p>(Example taken from <a href="https://developer.nvidia.com/blog/even-easier-introduction-cuda/" rel="noopener nofollow ugc">An Even Easier Introduction to CUDA | NVIDIA Technical Blog</a>)</p>

ptrblck: <p>You can find kernels in <code>aten/src/ATen/native</code>, such as e.g. the <a href="https://github.com/pytorch/pytorch/blob/99ddfaf572d152e1fa741fe0f546f742c6452b2e/aten/src/ATen/native/cuda/Dropout.cu#L47">fused dropout kernel</a>. Matmuls and other standard operations are used from cuBLAS(Lt) and other math libs.</p>
Curriculum Learning in torchRL?

Kevin_Fu: <p>Hi, I would like to implement a curriculum learning framework in torchRL. Specifically, I would like to increase the number of agents in an environment after a certain number of epochs or certain average reward reached in a navigation environment like torchRL’s <a href="https://pytorch.org/rl/reference/generated/torchrl.envs.libs.vmas.VmasEnv.html" rel="noopener nofollow ugc">VMASEnv</a>.</p>
<p>Is there a good way to do this within the framework?</p>

vmoens: <p>Happy to help you make that happen!</p>
<p>Let’s get technical about this: Can you first explain what you mean by increasing the number of agents? Does that mean that you have a dynamic action/observation space?</p>
<p>Also, is there anything you wish you could do but can’t at the moment with the primitives we propose?</p>
PyTorch's CUDA SVD function takes 3x the time of CuPy's

michaeleisel: <p>When running the following code, I found that PyTorch took ~9 seconds whereas CuPy took ~3 seconds. Setting the preferred linear algebra library to cuSOLVER didn’t seem to make a difference. Does anyone have any ideas why PyTorch is so much slower here, and what I might do to fix it?</p>
<pre><code class="lang-auto">import cupy
import torch
import time

def benchmark(name, f):
    f() # Warmup
    start = time.perf_counter()
    count = 0
    while True:
        f()
        curr = time.perf_counter()
        count += 1
        if curr &gt;= start + 1:
            result = count / (curr - start)
            print(f'{name}: {result:.3g}/sec')
            return

torch.set_default_dtype(torch.float64)
torch.set_default_device('cuda')
# torch.backends.cuda.preferred_linalg_library('cusolver')

size = 500
mat = torch.rand(size, size)
c_mat = cupy.random.rand(size, size, dtype=cupy.float64) # dtype is float64

def svd_torch():
    torch.linalg.svd(mat)
    cupy.cuda.Device().synchronize()

def svd_cupy():
    cupy.linalg.svd(c_mat)
    cupy.cuda.Device().synchronize()

benchmark('svd-torch', svd_torch)
benchmark('svd-cupy', svd_cupy)
</code></pre>

ptrblck: <p>It’s expected to see a slower performance using <code>float64</code> on the GPU and while I’m not deeply familiar with <code>cupy</code> it seems the CPU is used (based on the <code>dtype</code>). Using <code>float32</code> gives an expected speedup:</p>
<pre data-code-wrap="python"><code class="lang-python">svd-torch: 42.5/sec
svd-cupy: 9.54/sec
</code></pre>

michaeleisel: <p>Generally CuPy is on the GPU, and in fact in the docs for this method, it mentions that it calls cuSOLVER (<a href="https://docs.cupy.dev/en/stable/reference/generated/cupy.linalg.svd.html" class="inline-onebox" rel="noopener nofollow ugc">cupy.linalg.svd — CuPy 13.0.0 documentation</a>). When I run this myself for a 64-bit double matrix using cuSOLVER directly, with <code>cusolverDnDgesvd</code>, I get about 5 iterations per second. The difference between CuPy and this may be due to it using some other algorithm, e.g. gesvdj.</p>
Extracting output from intermediate layers in Inceptionv3

Rajnish_Gupta: <p>I am trying to extract output from the middle layer of Inception v3. I tried using</p>
<pre><code class="lang-auto">model = models.inception_v3(pretrained=True)
def get_features(name):
    def hook(model, input, output):
        features[name] = output.detach()

    return hook


model.global_pool.register_forward_hook(get_features("feats"))
</code></pre>
<p>although this produces an error:</p>
<p>“”"<br>
‘Inception3’ object has no attribute ‘global_pool’<br>
“”"</p>
<p>does anyone know some other way I can take out the features from this</p>

ptrblck: <p>As the error message explains, the model does not have a <code>.global_pool</code> attribute, so you might confuse it with <code>.avgpool</code>.</p>
BCELoss and CrossEntropyloss behaviour

Sai1: <p>Hi all, my problem is a binary class classification. when I’m training my model with BCELoss and sigmoid. I’m getting nan in second epoch right away. However, when I changed my loss to Crossentropy, the model is not outputting any nan’s. Can anyone explain what’s going on. if I should only be using BCELoss, can you suggest If I can find out what’s causing nan’s.<br>
Thank you</p>

ptrblck: <aside class="quote no-group" data-username="Sai1" data-post="1" data-topic="196727">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sai1/48/62072_2.png" class="avatar"> Sai1:</div>
<blockquote>
<p>if I should only be using BCELoss, can you suggest If I can find out what’s causing nan’s.</p>
</blockquote>
</aside>
<p>Remove the <code>sigmoid</code> and use <code>nn.BCEWithLogitsLoss</code> for better numerical stability and check if you would still see the NaN outputs.</p>
How to delete last linear layer of ViT-H or how to replace it with Identity()?

hasan11085: <p>Hello there! I’m trying to figure out how I could possibly replace or delete the last layer of <strong>VIT-H or ViT-b.</strong><br>
Here deleting children method doesn’t work like Resnet.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a2f495ea3735bed36d5d9c521edb0598b42c22e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5a2f495ea3735bed36d5d9c521edb0598b42c22e" title="Screenshot 2023-08-17 at 6.34.05 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2f495ea3735bed36d5d9c521edb0598b42c22e_2_620x500.jpeg" alt="Screenshot 2023-08-17 at 6.34.05 PM" data-base62-sha1="cROhzrzKi9EREJcEu9Tj7hW9UfQ" width="620" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2f495ea3735bed36d5d9c521edb0598b42c22e_2_620x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2f495ea3735bed36d5d9c521edb0598b42c22e_2_930x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2f495ea3735bed36d5d9c521edb0598b42c22e_2_1240x1000.jpeg 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-08-17 at 6.34.05 PM</span><span class="informations">1595×1285 342 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thanks in advance for the suggestion!</p>

hasan11085: <p>looking for an answer!</p>

ptrblck: <p>You can directly replace layers with an <code>nn.Identity</code> module.<br>
I would also not recommend bumping the thread after 2h of wait time, as it’s just spamming the board and not helpful.</p>

hasan11085: <p>I can access to model.encoder layers. But I didn’t find any solution how to access the last linear layer in VIT!</p>

danielmao2019: <p>If you are loading the vision transformer from <code>torchvision</code>, try modifying the linear layer by <code>model.heads = torch.nn.Identity()</code>. If you check the source code (I’m looking at version 0.15.2), you’ll see that the output of <code>model.encoder</code> is passed to <code>model.heads</code>.</p>
<p>In general, you can find out the attributes of a Python object by <code>obj.__dict__</code>.</p>
Real Time Inference Model dynamic determined.ai

Federico_Gemmo: <p>Hello! I have a question with respect to a behavior of python library to interact with determined.ai.<br>
Specifically, I want to be able to download a specific pytorch model from the model registry of determined.ai to do real-time inference. right now I can’t instantiate the MNIST trial class. determined.ai when I do the checkpoint download I get back a pickle file and a .pth file , being that I have to load the model, and determined.ai saves me under checkpoint/uuid_checkpoint, and I try to get the pickle or .pth file , it gives me : ValueError: Automatic import logic failed to import Trial class as <strong>main</strong>:MNistTrial. You will need to provide your Trial class via the trial_class argument intead.</p>
<p>but the trial class is static, in a real context this class I need dynamic, how do I do that?</p>

marksaroufim: <p>You will probably get better luck posting this on the determined github issues <a href="https://github.com/determined-ai/determined" class="inline-onebox">GitHub - determined-ai/determined: Determined is an open-source machine learning platform that simplifies distributed training, hyperparameter tuning, experiment tracking, and resource management. Works with PyTorch and TensorFlow.</a></p>
Overwriting/detaching part of a tensor

avi-amalanshu: <p>Hi, the situation is thus. Suppose I have 3 (say) models:</p>
<ol>
<li>two “guest” models which take batch inputs x1, x2 and output z1, z2 respectively</li>
<li>one “host” model which concatenates z = cat(z1, z2) and produces predictions y_hat</li>
</ol>
<p>each model has its own optimizer. In the training loop, I first obtain z1, then I obtain z2 and concatenate it to z1, and then I pass it through the host model. I calculate loss = some criterion(y, y_hat) and loss.backward().</p>
<p>I want to externally modify some elements of tensor z, say by setting some (predetermined) indices’ values to random values.</p>
<p>I suppose it isn’t possible to BP through this random node, so those elements need to be detached from the computational graph. But I want to update parameters that were not affected by this randomization. Is it possible to achieve this, or something to this effect?</p>
<p>Thanks!</p>
Contributing to Pytorch Documentation

maajidkhann: <p>Hello Community, we have recently merged our code to Pytorch (<a href="https://github.com/pytorch/pytorch/pull/115184" class="inline-onebox" rel="noopener nofollow ugc">Disables denormal floating numbers on ARM CPU by maajidkhann · Pull Request #115184 · pytorch/pytorch · GitHub</a>) and we are yet to update the documentation. Just checking how to create a pull request to update documentation?</p>
<p><strong>This is the page we are looking to update:</strong><br>
<a href="https://pytorch.org/docs/stable/generated/torch.set_flush_denormal.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.set_flush_denormal.html</a></p>
<p>Is there any branch in Pytorch github, where we can update? Please assist. Thanks.</p>
How can I prevent TorchScript from including the original Python source code?

michaeleisel: <p>I’ve created a model via <code>torch.jit.script(...).save('file.pt')</code> in Python, and want to export it without including the original Python source code at all. This is because the model is going to be deployed on-prem for external clients, and I’d like to make it a bit less trivial for clients to see how the underlying model works. Is there any way to not include the original Python source code in the .pt file? Also, is there any other way to further obfuscate the TorchScript? I know there are Python obfuscation tools, such as pyarmor, but I wonder if they’d interfere with optimizations, and also really put TorchScript’s ability to interpret Python correctly to the test.</p>