akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Performance Drop in Stereo Matching Network After PyTorch Version Update

sleepat-11: <p>I have been training the same stereo matching network with different versions of PyTorch and noticed a significant performance decrease with PyTorch 1.13 compared to PyTorch 1.2. According to the original author’s response, this is due to differences in the implementation of the <code>interpolate</code> function between versions. Specifically, my network contains an operation like this:</p>
<p>python</p>
<pre><code class="lang-auto">F.interpolate(inputdata, [d, h, w], mode='trilinear', align_corners=False)
</code></pre>
<p>Where <code>inputdata</code> is a five-dimensional tensor. I would like to implement the <code>interpolate</code> function from PyTorch 1.2 myself, but upon inspecting the code, I found that <code>torch._c</code> has been encapsulated and is inaccessible.</p>
<p>Does anyone have advice on how I could proceed to either access this function or an alternative way to replicate the PyTorch 1.2 behavior? Your assistance would be greatly appreciated. Thank you!</p>
MLP Activation checkpointing

Luosuu: <p>Hi,</p>
<p>I am trying to learn how to use activation checkpointing provided by <code>torch.utils.checkpoint</code> and <code>torch.distributed.algorithms._checkpoint.checkpoint_wrapper</code>. So I have a script to apply checkpointing to a feedforward network to profile the memory usage at different steps:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.utils.checkpoint as checkpoint

from torch.distributed.algorithms._checkpoint.checkpoint_wrapper import (
        apply_activation_checkpointing,
        checkpoint_wrapper,
        CheckpointImpl,
)

class FeedForward(nn.Module):
    def __init__(self, hidden_size, intermediate_size, num_layers):
        super(FeedForward, self).__init__()
        self.layers = nn.ModuleList([
            nn.Sequential(
                nn.Linear(hidden_size, intermediate_size),
                nn.GELU(),
                nn.Linear(intermediate_size, hidden_size)
            ) for _ in range(num_layers)
        ])

    def forward(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

print(torch.__version__)

hidden_size = 1024
intermediate_size = 4096
num_layers = 8
sequence_length = 1024
batch_size = 128

# Baseline model without checkpointing
torch.cuda.reset_peak_memory_stats()
with torch.autograd.profiler.profile(profile_memory=True, use_cuda=True) as prof:
    print("model with checkpointing wrapper")
    model = FeedForward(hidden_size, intermediate_size, num_layers).cuda()
    optimizer = torch.optim.Adam(model.parameters())
    model = checkpoint_wrapper(model) # comment this line to disable checkpointing
    model.train()

    print("Peak memory usage after model init: ", torch.cuda.max_memory_allocated() / 1024 / 1024, "MB")
    input_data = Variable(torch.randn(batch_size, sequence_length, hidden_size), requires_grad=True).cuda()
    print("Peak memory usage after data init: ", torch.cuda.max_memory_allocated() / 1024 / 1024, "MB")
    output = model(input_data)
    print("Peak memory usage after forward: ", torch.cuda.max_memory_allocated() / 1024 / 1024, "MB")
    optimizer.zero_grad()
    output.sum().backward()
    optimizer.step()
    print("Peak memory usage after backward: ", torch.cuda.max_memory_allocated() / 1024 / 1024, "MB")

</code></pre>
<p>and I got output when I enable the checkpointing</p>
<pre><code class="lang-auto">Peak memory usage after model init:  256.15625 MB
Peak memory usage after data init:  768.15625 MB
Peak memory usage after forward:  5384.28125 MB
Peak memory usage after backward:  40736.4072265625 MB
</code></pre>
<p>when I disable the checkpointing:</p>
<pre><code class="lang-auto">Peak memory usage after model init:  256.15625 MB
Peak memory usage after data init:  768.15625 MB
Peak memory usage after forward:  37641.28125 MB
Peak memory usage after backward:  40224.4072265625 MB
</code></pre>
<p>where I see the peak memory usage after backward is even higher when checkpointing enabled!</p>
<p>How should I fix it?</p>
My custom loss function would not work

zfan: <p>Hi, I created a custom loss function in which the loss is calculated based on silhouette score. Here is my loss function:</p>
<pre><code class="lang-auto">class ClusteringLoss(torch.nn.Module):
    def __init__(self):
        super(ClusteringLoss,self).__init__()

    def forward(self, embeddingData, origData, k):
	
        repData = embeddingData.detach().numpy();
        
        kmeansModel = KMeans(n_clusters=k).fit(repData);
        yhat = kmeansModel.labels_;
        
        silhouetteScore = metrics.silhouette_score(origData, yhat);
        print("SilhouetteScore1: {}".format(silhouetteScore));
        
        loss = Variable( torch.tensor(silhouetteScore), requires_grad = True);
        loss = alpha*(1-loss);
		
        return loss;
</code></pre>
<p>The argument <code>embeddingData</code> is the output from the last layer of the model and it is a part of the gradient flow. I needed to use it to run Kmeans clustering and then calculate the silhouette score. Since the computed silhouette score can indicate how well the clustering result is, so I used it as the loss for my loss function.<br>
However, I noticed that the gradients would not be updated after running a few epochs. I think the gradient flow was cut off, but I do not know which part of the code causes the problem. Any help will be appreciated. Thank you so much.</p>

ptrblck: <p>You are explicitly detaching the output of your model, which will cut the computation graph and will thus explain why your model does not learn anything.<br>
Besides that you are also using 3rd party libraries (in this case <code>numpy</code> and <code>scikit-learn</code>), which Autograd also won’t be able to understand. You would either have to implement the used methods in PyTorch directly or a custom <code>autograd.Function</code> where the <code>forward</code> and <code>backward</code> need to be implemented manually.<br>
Also, wrapping the already detached tensor into the deprecated <code>Variable</code> will not reattach it to the computation graph.</p>

zfan: <p>Thank you so much for taking the time to respond to my post. I appreciate it. It is so nice of you to do that. Thank you so much.</p>
Multi-thread usage of open_clip

Mobin_Esmaily: <p>Hi, I use open_clip with a pre-trained model on my fast Api server to compare the similarity of images, but after a lot of concurrent http requests to the server, it doesn’t run in parallel, it runs slower and slower. Please help me about it thanks</p>
<pre><code class="lang-auto">device = "cuda" if torch.cuda.is_available() else "cpu"
model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-16-plus-240', pretrained="laion400m_e32",device=device)

def imageEncoder(img):
    img1 = preprocess(img).unsqueeze(0).to(device)
    img1 = model.encode_image(img1)
    return img1


def generateScore(image1, image2):
    img1 = imageEncoder(image1)
    img2 = imageEncoder(image2)
    cos_scores = util.pytorch_cos_sim(img1, img2)
    score = round(float(cos_scores[0][0]) * 100, 2)
    return score



class Imagesclass(BaseModel):
    f_base64: str
    s_base64: str

@app.post("/cmp")
async def createcmp(imagelcass: Imagesclass):

    image_bytesone = base64.b64decode(imagelcass.f_base64)
    pil_image1 = Image.open(io.BytesIO(image_bytesone))
	
	mage_bytestwo = base64.b64decode(imagelcass.s_base64)
    pil_image2 = Image.open(io.BytesIO(mage_bytestwo))
	
	score = generateScore(pil_image1, pil_image2)
	
	
    return JSONResponse(status_code=status.HTTP_200_OK, content={"status": "ok", "score": str(score)})
</code></pre>

Mobin_Esmaily: <p>i really need help about this problem.<br>
thank you.</p>

Mobin_Esmaily: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I need your help please.</p>
Parallelizing loop over modules

Julian_Buchel: <p>Hi,</p>
<p>I have a post-step hook attached to my optimizer that loops over each linear layer in the model. For each linear layer, it calls a method called <code>clip_weights</code>. I feel like doing this sequentially can be quite expensive when the number of layers become high (in the 500s), and the layers also become large.</p>
<p>What would be the best approach to parallelize this? Each <code>clip_weights</code> is still intensive as it calculates the <code>std</code> and <code>clamp</code>s on each weight, so I guess it has to be done in GPU.</p>
<p>Here is a rough sketch of what I was thinking about:</p>
<pre><code class="lang-auto">import torch.multiprocessing as mp
from torch.multiprocessing import Pool

def worker(layer):
    layer.clip_weights()

def parallel_clip_weights(layers_gen):
    with Pool(processes=mp.cpu_count()) as pool:
        # Use imap or imap_unordered
        # imap returns the results in the order the inputs were given
        # imap_unordered returns the results as soon as they are ready, not necessarily in order
        for _ in pool.imap(worker, layers_gen):
            pass

# Assuming layers is a generator
# layers_gen = model.layers()
parallel_clip_weights(layers_gen)
</code></pre>
<p>But I am not sure what the best approach is.<br>
Any suggestions?</p>
Kill job if exception raised during NCCL AllReduce

bshama: <p>Hi!</p>
<p>I’m currently training using pytorch distributed with NCCL_ASYNC_ERROR_HANDLING and often run into an issue where one of the training ranks throws an exception but the job won’t fail until our configured NCCL timeout has been reached. I’d like to be able to immediately kill all ranks/processes when an exception is thrown by one of the ranks, is there a good way to accomplish this?</p>
<p>Thanks!</p>

bshama: <aside class="quote no-group" data-username="bshama" data-post="1" data-topic="198459">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bshama/48/68295_2.png" class="avatar"> bshama:</div>
<blockquote>
<p>I’m currently training using pytorch distributed with NCCL_ASYNC_ERROR_HANDLING and often run into an issue where one of the training ranks throws an exception but the job won’t fail until our configured NCCL timeout has been reached. I’d like to be able to immediately kill all ranks/processes when an exception is thrown by one of the ranks, is there a good way to accomplish this?</p>
<p>Thanks!</p>
</blockquote>
</aside>
<p>Resolved this by changing the way we kick off our torch jobs and changing our k8s jobs to kill the job if a single pod fails.</p>
Replicating JAX style code in PyTorch

Matthew_Retchin: <p>Are there any advantages to using JAX now that torch.func and torch.compile exist? In other words, what parts of PyTorch are intended to be used together to recreate a JAX-like experience, if that’s possible? There are many compilation options available, and I’d appreciate a clear and succinct answer on the currently recommended option</p>
<p>I’m aware that JAX is intended to replicate NumPy’s API as much as possible, which is not in PyTorch’s scope. I’m seeking a head-to-head comparison without an unrealistic expectation of the possibility of stretching PyTorch’s shape to fit JAX exactly.</p>
<p>If PyTorch currently falls short (for instance, how PyTrees are handled with vmap and gradient), please enumerate the ways so I can make an informed choice about what to use. I’m particularly interested in the debugging story when it comes to compiled code as well as compilation speed and ultimate performance of compiled code relative to JAX.</p>
<p>I’m especially interested in the opinion of someone who’s extensively used both the newer PyTorch features and JAX.</p>
Effishent matmul in fused kernals

neva: <p>I am looking to fuse a dequntization into a mathmul operation. now I could always do 1 then the other but I would really like to have it on the same kernel so that there is no need to store weights on global memory.</p>
<p>is there a way to take the existing matmul but run it inside of the kernel? or put a preprocessing step thats inlined to the cuda kernel.</p>
File Not Found in `sem_unlink`

Kore_ana: <p>I got the following error while running DDP. Although the code seems to run well, I get the following in the error log. (which seems to be error that are ignored) Should I be worried?</p>
<pre><code class="lang-auto">Exception ignored in: &lt;Finalize object, dead&gt;
Traceback (most recent call last):
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/synchronize.py", line 87, in _cleanup
    sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Exception ignored in: &lt;Finalize object, dead&gt;
Traceback (most recent call last):
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/synchronize.py", line 87, in _cleanup
    sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Exception ignored in: &lt;Finalize object, dead&gt;
Traceback (most recent call last):
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/synchronize.py", line 87, in _cleanup
    sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Exception ignored in: &lt;Finalize object, dead&gt;
Traceback (most recent call last):
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/synchronize.py", line 87, in _cleanup
    sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Exception ignored in: &lt;Finalize object, dead&gt;
Traceback (most recent call last):
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/synchronize.py", line 87, in _cleanup
    sem_unlink(name)
FileNotFoundError: [Errno 2] No such file or directory
Exception ignored in: &lt;Finalize object, dead&gt;
Traceback (most recent call last):
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/util.py", line 224, in __call__
    res = self._callback(*self._args, **self._kwargs)
  File "/home/connectome/dyhan316/.conda/envs/VAE_3DCNN_older_MONAI/lib/python3.8/multiprocessing/synchronize.py", line 87, in _cleanup

</code></pre>

zuliani99: <p>I have the same warning, have you solved it?</p>
Torchvision fails to load LSUN dataset

CalinTimbus: <p>Hello everyone.<br>
I am trying to load the LSUN dataset.<br>
However, the following code snippet does not work:</p>
<p>*<strong>dataset = torchvision.datasets.LSUN(root=’./data’, classes = [‘bedroom_train’], transform = transform)</strong> , giving me the error:</p>
<p>*<strong>Error: ./data/bedroom_train_lmdb: The system cannot find the path specified.</strong></p>
<p>I assume there is a problem with the path given in the root, can anyone please shed some light on this problem?<br>
Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>PyTorch version used : 0.4.1 (latest stable release)<br>
Torchvision verison used: 0.2.1</p>

Deeply: <p>try <code>'/data'</code></p>
<p>if it does not work, try the full path of your data</p>
<p><code>'/home/ClainTimbus/your_app/data'</code></p>
<p>the above works for me.</p>

CalinTimbus: <p>By full path I assume full absolute path.<br>
Thank you, I will try. As soon as I get an answer/solution I will inform you of it.</p>

CalinTimbus: <p>It doesn’t work for me. I also need to specify that I’m on Windows, not on Ubuntu.</p>

cutezhaoooo: <p>Hello, I encountered the same problem. Have you solved it?</p>
How to train my model on multiple GPU

Shrijayan: <p>I am trying to train the Sentence Transformer Model named cross-encoder/ms-marco-MiniLM-L-12-v2 where When I try to train it utilizes only one GPU, where in my machine I have two GPUs. I tried DataParallel and DistributedDataParallel, but didn’t Work.</p>
<pre><code class="lang-auto">from sentence_transformers import SentenceTransformer, losses
from torch.utils.data import DataLoader

# Replace 'model_name' and 'max_seq_length' with your actual model name and max sequence length
model_name = 'your_model_name'
max_seq_length = your_max_seq_length

# Load SentenceTransformer model
model = SentenceTransformer(model_name)
model.max_seq_length = max_seq_length

# Replace 'train_examples' with your actual training examples
train_examples = your_train_examples

# Create DataLoader for training
train_dataloader = DataLoader(train_examples, batch_size=16, drop_last=True, shuffle=True)

# Define the loss function
train_loss = losses.MarginMSELoss(model)

# Tune the model
model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=500, warmup_steps=int(len(train_dataloader) * 0.1))

# Replace 'output_path' with the desired path for saving the trained model
output_path = 'your_output_path'

# Save the model after training
model.save(output_path)
</code></pre>
<p>I am trying to train the Sentence Transformer Model named cross-encoder/ms-marco-MiniLM-L-12-v2 where When I try to train it utilizes only one GPU, where in my machine I have two GPUs. I tried DataParallel and DistributedDataParallel, but didn’t Work.</p>
<pre><code class="lang-auto">from sentence_transformers import SentenceTransformer, losses
from torch.utils.data import DataLoader

# Replace 'model_name' and 'max_seq_length' with your actual model name and max sequence length
model_name = 'your_model_name'
max_seq_length = your_max_seq_length

# Load SentenceTransformer model
model = SentenceTransformer(model_name)
model.max_seq_length = max_seq_length

# Replace 'train_examples' with your actual training examples
train_examples = your_train_examples

# Create DataLoader for training
train_dataloader = DataLoader(train_examples, batch_size=16, drop_last=True, shuffle=True)

# Define the loss function
train_loss = losses.MarginMSELoss(model)

# Tune the model
model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=500, warmup_steps=int(len(train_dataloader) * 0.1))

# Replace 'output_path' with the desired path for saving the trained model
output_path = 'your_output_path'

# Save the model after training
model.save(output_path)
</code></pre>
What happens when we don't set padding_idx?

wenlongzhao094: <p>What are the usage of padding_idx for nn.Embedding? It seems that the vector at padding_idx will be initialized as zeros. Also this vector is not trained. But when is it used for padding?</p>
<p>What happens when we do not set a padding_idx?<br>
What happens when we do not set a padding_idx but still have  at the beginning of the vocab?</p>

ptrblck: <p>If <code>padding_idx</code> is set, the output tensor will contain all zeros at the position, where the input tensor had the <code>padding_idx</code>.<br>
You could use it to ignore specific inputs without the need to remove these index from your input tensor.</p>

gat1s: <p>Hello,</p>
<p>I think there is still an ambiguity in the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html" rel="noopener nofollow ugc">documentation</a>, which states that “the entries at <code>padding_idx</code> do not contribute to the gradient; therefore, the embedding vector at <code>padding_idx</code> is not updated during training”.</p>
<p>Here is a toy example in which I sum the entries of the embedding matrix and backpropagate from there:</p>
<pre><code class="lang-auto">a = nn.Embedding(4, 4, padding_idx = 0)

S = a.weight.sum()

S.backward()

a.weight.grad
tensor([[1., 1., 1., 1.],
        [1., 1., 1., 1.],
        [1., 1., 1., 1.],
        [1., 1., 1., 1.]])

optimizer = optim.SGD(a.parameters(), lr=0.1)

optimizer.step()

a.weight
Parameter containing:
tensor([[-0.1000, -0.1000, -0.1000, -0.1000],
        [-1.2080, -0.6763, -0.1524, -2.1535],
        [-0.0298, -1.2028,  1.4500, -0.8885],
        [-0.8094, -0.3046, -2.0699, -0.7560]], requires_grad=True)
</code></pre>
<p>I see that the entry at the PAD index does develop a gradient and does update.</p>
<p>Could you explain how to understand the documentation? (Or what I’ve done wrong?)</p>

Nils_Smitham86: <p>you should not use embedding weight straightly, in your code, you treate embedding weights as a ordinary tensor, but not nn.Embedding() layer or nn.Embedding().forward() function</p>
Runtime error: target_lengths must be of size batch_size with CTC loss using a batch size of 1

Percibus: <p>I’m trying to implement a simple transformer using CTC loss. I’m using a batch size of 1. My inputs are masked as follows:</p>
<pre><code class="lang-auto">        mask = ~mask.any(axis=2)
        mask = mask.unsqueeze(-1).expand(-1, -1, self.embedding_dim).bool()
        
        positional_encodings = get_positional_encoding(video_frames.size(1), self.embedding_dim).to(video_frames.device)
        video_frames = video_frames + positional_encodings
            
        masked_video_frames = video_frames * mask.float()
               
        transformer_output = self.transformer_encoder( # expects [batch, seq_len, emb_dim]
            masked_video_frames,
        ) 
        letter_predictions = self.ctc_head(transformer_output)
        return letter_predictions, mask
</code></pre>
<p>When computing the CTC loss on the dataloader loop like follows:</p>
<pre><code class="lang-auto">        video_frames, target, mask = data_batch 
        optimizer.zero_grad()    

        letter_predictions, frame_mask = model(video_frames.to(device), mask.to(device))
        log_probs = nn.functional.log_softmax(letter_predictions, dim=2) 
        
        # Compute CTC loss
        frame_mask = frame_mask.any(dim=2).unsqueeze(-1)  
        masked_log_probs = log_probs * frame_mask.float()  
        
        input_lengths = torch.sum(~frame_mask, dim=0)  
        target_concatenated = torch.cat(target, dim=0).to(device)         
        target_length = len(target_concatenated)  
        target_lengths = torch.tensor([target_length], dtype=torch.long).to(device)
        
        print(f"Log probs shape: {masked_log_probs.shape}")
        print(f"Input lengths shape: {input_lengths.shape}")
        print(f"Target concatenated: {target_concatenated}")
        print(f"Target concatenated shape: {target_concatenated.shape}")
        print(f"Target lengths shape: {target_lengths.shape}")
       
        loss = criterion(masked_log_probs, target_concatenated, input_lengths, target_lengths)
</code></pre>
<p>I got: “RuntimeError: target_lengths must be of size batch_size”.</p>
<p>Now my shapes are as follows:</p>
<blockquote>
<p>Log probs shape: torch.Size([1, 152, 58])<br>
Input lengths shape: torch.Size([152, 1])<br>
Target concatenated: tensor([21, 23, 17], device=‘cuda:0’)<br>
Target concatenated shape: torch.Size([3])<br>
Target lengths shape: torch.Size([1])</p>
</blockquote>
<p>I’m stuck with this, sorry if is too beginner question but can’t found what is happening. My target lenghts shape is already 1, same as my batch size.</p>
<p>Thanks in advance for any help.</p>

Percibus: <p>I was passing log probs as a tensor of shape [batch_size, sequence_length, features], while the CTC loss expects the batch size on the second dimension.</p>
Use dropout from specific score on train/validation

kfir7755: <p>Hi, I have a problem with my model due to low amounts of data.<br>
If I use dropout layer, sometimes it is unable to learn and get stucked on local minimum of the loss.<br>
If I don’t use it, the model successfully learns all the time but there is overfit.<br>
Basically, I want to use the dropout layer after I made sure the model has surpassed the local minimum that caused me those problems.<br>
I thought about the following</p>
<pre><code class="lang-auto">def forward(self,x,score=None):
    x = self.seq1(x) # each seq has layers,activation functions and batchNorm
    if score is None or (score is not None and score &gt; threshold):
        x = self.dropout(x)
    x = self.seq2(x)
    return x
</code></pre>
<p>Do you think it is a valid way to do so? will it cause me problems with the torch autograd or optimizer.step()?<br>
if you can come up with a better way of doing so, please let me know.<br>
Thank you in advance,<br>
Kfir</p>
RuntimeError: expand(torch.FloatTensor{[1, 8, 263, 4]}, size=[8, 263, 4]): the number of sizes provided (3) must be greater or equal to the number of dimensions in the tensor (4)

Matt_Morency: <pre><code class="lang-auto"> max_len = max(len(bbox) for bbox in bbox_list)
    padded_bbox_list = [bbox + [[0, 0, 0, 0]] * (max_len - len(bbox)) for bbox in bbox_list]
    bbox_tensor = torch.tensor(padded_bbox_list, dtype=torch.float32)

    
    bbox_tensor = bbox_tensor.unsqueeze(0)
    bbox_tensor = bbox_tensor.expand(encoded_inputs['input_ids'].shape[0], max_len, 4)

</code></pre>
<p>runtimeerror: expand(torch.floatTensor{[1,8,263,4]}, size=[8,263,4]):</p>
Do PyTorch transformers split the feature dimension?

alaria: <p>I read that the PyTorch implementation of the multihead attention actually splits the features by the head count. Would that not mean that communication between these features is impossible and that using more than one head could present a significant bottleneck?</p>

vdw: <p>This is the approach proposed in the Transformer in the original paper <a href="https://arxiv.org/abs/1706.03762" rel="noopener nofollow ugc">“Attention Is All You Need”</a>. Their motivation is that the total number of parameters should not depend on the number of heads – at least not without also increasing the core model size.</p>
<p>I’m not quite sure what you mean by the communication of these features. The purpose of multiple heads is to some extend to be independent to potentially pick up on different patterns/aspects in the data.</p>

alaria: <p>Would that not imply that the end result of the attention layer potentially can’t determine correlations between two features? Or at least can’t measure their importances against each other?</p>
Original Encoder-Decoder Transformer: Text Generation?

Imahn: <p>In order to translate a sentence with the original encoder-decoder transformer, the following happens:</p>
<ol>
<li>The source sentence is encoded,</li>
<li>the decoder initially gets as input a start token, learns to generate a new token based on the previously seen tokens, which is concatenatted with the decoder input tokens.</li>
</ol>
<p>Let’s say we want to translate the sentence “Wie geht es dir?”<br>
This sentence would be fed into the encoder (of course first tokenized), and the decoder gets as input the start token <code>[SOS]</code> in shape <code>(1, 1)</code>.  There’s now a multi-head attention block that whose queries and keys come from the encoder output in shape <code>(1, num_heads, seq_length, embed_dim)</code> and the values come from the previous masked multi-head attention block in shape <code>(1, num_heads, 1, embed_dim)</code> (so effectively, when we feed the token <code>[SOS]</code> into the decoder, we have a sequence length of <code>1</code>).</p>
<p>In self-attention, we calculate <code>softmax(QK^T / sqrt(d_v))</code>, which has a resulting shape of <code>(1, num_heads, seq_length, seq_length)</code>, but now the multiplication with <code>V</code> would fail and result in a shape mismatch?</p>

vdw: <p>Maybe <a href="https://github.com/chrisvdweth/nus-cs4248x/blob/master/3-neural-nlp/Section%204.2%20-%20Transformer%20Machine%20Translation.ipynb" rel="noopener nofollow ugc">this notebook</a> might help. It is a extended version of an official PyTorch tutorial. Or maybe <a href="https://github.com/chrisvdweth/nus-cs4248x/blob/master/3-neural-nlp/Section%204.1%20-%20Transformer%20(Basic%20Architecture).ipynb" rel="noopener nofollow ugc">this notebook</a>, going through the Transformer implementation using an from-scratch implementation. If nothing else, you can add print statements to check all the relevant tensor shapes</p>
Canny kornia RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.HalfTensor) should be the same

Samuel2: <p>I’m trying to modify a module in YoloV9, but when I use the Canny function from Kornia. I got this error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd8c5b7bc75b1daf2ef685cc52b4c3d905a71364.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cd8c5b7bc75b1daf2ef685cc52b4c3d905a71364" title="WhatsApp Image 2024-03-08 at 20.15.11_cdc2122e"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd8c5b7bc75b1daf2ef685cc52b4c3d905a71364_2_553x500.jpeg" alt="WhatsApp Image 2024-03-08 at 20.15.11_cdc2122e" data-base62-sha1="tkmAVQrVWRdtpdrQykGr5C85rUg" width="553" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd8c5b7bc75b1daf2ef685cc52b4c3d905a71364_2_553x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd8c5b7bc75b1daf2ef685cc52b4c3d905a71364.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd8c5b7bc75b1daf2ef685cc52b4c3d905a71364.jpeg 2x" data-dominant-color="E8E8E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WhatsApp Image 2024-03-08 at 20.15.11_cdc2122e</span><span class="informations">711×642 164 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Here is my code :</p>
<pre><code class="lang-auto">class SpectralFeatureAdaptation(nn.Module):
    def __init__(self,c1):
        super().__init__()
        self.conv1= nn.Conv3d(4, 16, 3, stride=1, padding='same')  ##in=4 if edge info used
        self.conv2= nn.Conv3d(4, 16, (3,5,5), stride=1, padding='same') ##in=4 if edge info used
        self.conv3= nn.Conv3d(4, 16, (3,7,7), stride=1, padding='same') ##in=4 if edge info used
        self.conv4= nn.Conv3d(48, 3, 3, stride=1, padding='same')
   
    def forward(self, input):
        mag, edge = canny(input)
        edge_fused=torch.cat((input,edge), 1)
        ba,ch,dim1,dim2=list(edge_fused.size())
        input=input.reshape(ba,ch,-1,dim1,dim2)
        x1= self.conv1(edge_fused)
        x2= self.conv2(edge_fused)
        x3= self.conv3(edge_fused)
        x4=torch.cat((x1,x2,x3), 1) ## concatinate feature map
        x5= self.conv4(x4)
        ba,ch,di,dim1,dim2=list(x5.size())
        x5=x5.reshape(ba,ch,dim1,dim2)
        return x5

</code></pre>

ptrblck: <p>Based on the error message it seems <code>canny</code> contains <code>float16</code> parameters while <code>input</code> is in <code>float32</code> raising the <code>dtype</code> mismatch. You could add a <code>torch.cuda.amp.autocast</code> context to this operation in case you want to use mixed-precision execution.</p>
<p>Also, it’s better to post code snippets instead of screenshots as they are not only hard to read but also cannot be indexed into our search. <code>canny</code> is btw. also not defined, so I don’t know how its <code>dtype</code> is defined.</p>
Dataset interaction with a global array

srikarym: <p>Can the <strong>getitem</strong> function of a dataset class write to a global array while iterated by a dataloader?</p>
<pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader

indexes_seen = []


class MyDataset(Dataset):
    def __init__(self,):
        
        print("constructor")
        self.dummy = list(range(100))

    def __len__(self):
        return len(self.dummy)

    def __getitem__(self,idx):
        global indexes_seen
        indexes_seen.append(idx)     
        
        return self.dummy[idx]

ds = MyDataset()
dl = DataLoader(ds, batch_size=4, num_workers=4, shuffle=False)

for item in dl:
    break

print(indexes_seen)

&gt;&gt;&gt; [ ]

</code></pre>
<p>indexes_seen array is printed as empty, but if printed in the <strong>getitem</strong> function, I can see elements being added to it.</p>

ptrblck: <p>You might need to share the actual memory as seen e.g. in <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py">this example</a>.</p>
Save BatchedTensor to a pickle file

mfkasim: <p>I would like to save my <code>BatchedTensor</code> (the tensor type that’s produced by <code>functorch.vmap</code>) to a file for debugging my application. However, a RuntimeError is raised whenever I call <code>torch.save</code> to the BatchedTensor: <code>RuntimeError: Cannot access data pointer of Tensor that doesn't have storage</code>.</p>
<p>How can I save a <code>BatchedTensor</code>?</p>

ptrblck: <p>Could you share a minimal and executable code snippet returning the <code>BatchedTensor</code> object as I’m seeing plain tensors while running the <a href="https://pytorch.org/tutorials/prototype/vmap_recipe.html"><code>vmap</code> tutorial</a>.</p>

mfkasim: <p>Here is the minimal executable code:</p>
<pre><code class="lang-python">import torch
import functorch

def func(x: torch.Tensor) -&gt; torch.Tensor:
    # some function where we want to debug closely here
    y = 2 * x
    print(x, y)  # to show x &amp; y are BatchedTensors
    # try to save the tensors (the error occurs here)
    torch.save((x, y), "somefile.pt")
    return y

x = torch.randn((4, 5))
y = functorch.vmap(func)(x)
</code></pre>

ptrblck: <p>Thanks for the code snippet.<br>
I don’t know if there is a public and supported way to save the internal and intermediate tensors, but this code using internal calls should work:</p>
<pre><code class="lang-python">def func(x: torch.Tensor) -&gt; torch.Tensor:
    # some function where we want to debug closely here
    y = 2 * x
    print(x, y)  # to show x &amp; y are BatchedTensors
    # try to save the tensors (the error occurs here)
    vx = torch._C._functorch.get_unwrapped(x)
    vy = torch._C._functorch.get_unwrapped(y)
    torch.save((vx, vy), "somefile.pt")
    return y

x = torch.randn((4, 5))
y = functorch.vmap(func)(x)
</code></pre>
<p>Note that these calls can easily break since they are internal and no backwards-compatibility is guaranteed. <a class="mention" href="/u/richard">@richard</a> might know more about this use case.</p>

mfkasim: <p>Thanks, <a class="mention" href="/u/ptrblck">@ptrblck</a>! The function <code>get_unwrapped</code> is very helpful in debugging within vmapped-function. It would be useful if there’s a public API for this kind of function.</p>

ptrblck: <p>Yeah, I’ll let <a class="mention" href="/u/richard">@richard</a> add a comment here in case this is planned as a future debugging tool (or maybe there is another already public API that I’m missing).</p>

zhang0: <p>how i solve the problem?<br>
the function is work in torch.vmap,  i got the error<br>
File “/home/zhangxiao/work/JAX-ReaxFF/torchreaxff/reaxffpotential.py”, line 933, in calculate_covbon_pot<br>
print(“TP:”,vx.detach().cpu().numpy())<br>
^^^^^^^^^^^^^^^^^^^^^^^^^<br>
RuntimeError: Cannot access data pointer of Tensor that doesn’t have storage</p>
Using nn.Sequential to use models.video.mvit_v2_s for feature extraction

Muhammad_Umar_Farooq: <p>Hi,<br>
I’m trying to replace mc3_18 with mvit_v2_s from torchvision.models.video but getting tensor shape error. Here is my code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torchvision import models

# base = models.video.mc3_18(weights='DEFAULT', progress=True)
base = models.video.mvit_v2_s(weights='DEFAULT', progress=True)
base = nn.Sequential(*list(base.children())[:-1])

batch_size = 4  # Adjust as needed
dummy_input = torch.randn(batch_size, 3, 16, 224, 224)
output = base(dummy_input)
print("Model input shape:", dummy_input.shape)
print("Model output shape:", output.shape)

</code></pre>
<p>Without applying nn.Sequential on all the layers except the last, mvit_v2_s model works. Ideally it should work as mc3_18 is working with nn.Sequential and give the out with dim <code>[4, 512, 1, 1, 1]</code>. Using nn.Sequential with mvit_v2_s gives this error that I’m unable to debug.</p>
<pre><code class="lang-auto">def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
    411         class_token = self.class_token.expand(x.size(0), -1).unsqueeze(1)
--&gt; 412         x = torch.cat((class_token, x), dim=1)
    413 
    414         if self.spatial_pos is not None and self.temporal_pos is not None and self.class_pos is not None:

RuntimeError: Tensors must have same number of dimensions: got 3 and 5
</code></pre>
<p>Is there any other way to use mvit_v2_s for features extraction?</p>

ptrblck: <p>Putting all layers into an <code>nn.Sequential</code> container might be tricky, as the original <code>forward</code> method is not trivial as seen <a href="https://github.com/pytorch/vision/blob/423a1b0ebdea077cc69478812890845741048d2e/torchvision/models/video/mvit.py#L545-L565">here</a>. E.g. you would need to make sure the used functional API calls are also added, e.g. the <code>x.flatten(2)</code> or <code>.transpose()</code> calls as well as the slicing etc.<br>
If you want to extract features, you might want to use forward hooks instead.</p>

Muhammad_Umar_Farooq: <p>Ok thanks, will use forward hooks!</p>