akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Memory fills up gradually over a few iterations

username: sajangohil
cooked: <p>I am training a huggingface <a href="https://huggingface.co/docs/transformers/main/model_doc/layoutlmv2" rel="noopener nofollow ugc">LayoutLMV2</a> model and I have a custom Dataset class that I call using a dataloader with batch/random sampler.<br>
My dataset loads all relevant info that can stay in memory (labels, paths, etc) and all the main processing happpens in <code>__getitem__</code> function for relevant index of data (Loading image, calling the huggingface tokenizer/processor etc).</p>
<p>If my dataset is smaller (few thousand data points), things work normally, I see almost no change in RAM used, but for large data (30-40 thousand files), loading the initial data works fine, only difference is the RAM consumed (more consumed as more datapoints) but while iterating over the dataloader, the RAM consumption <strong>gradually increases</strong>.</p>
<p>I have removed the model from the pipeline and the processor in <code>__getitem__</code> (I basically just loop over the dataloader) and this still happens till the process gets killed because memory gets filled. My training happens on GPU and GPU memory stays normal while the RAM fills up. This happens over a few thousand steps, which doesn’t happen with smaller datasets.</p>
<p>What could be the reason for this? What could be a way around this?</p>
<p>I will see if I can share a reproducible example.</p>

username: amirshamaei
cooked: <p>I experience the same issue. I am using pytorch lightning framework.</p>
Get Filenames after Random Split (ImageFolder)

username: quirlescence
cooked: <p>Hi,</p>
<p>I am using torchvision ImageFolder to load my data and have two questions. Maybe you can help me.</p>
<p>After loading, I perform a random split for training, validation and test. How can I get access to a list of the filenames in each dataloader after the split? I would like to get a list of the path/filenames so that I know which image is in which subset. Is this possible without writing a custom dataset?</p>
<p>Thank you much!!</p>
Current epoch access in Learning rate scheduler

username: cbd
cooked: <p>I want to customize the learning rate scheduler for 20 epochs. For first 10 epochs constant learning rate and for last 10 epochs exponential decreasing rate. I want something like below in “get_scheduler(optimizer,opt)”.</p>
<pre><code class="lang-auto"> if(epoch&lt;11):
        scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=1)
 else:        
        scheduler = lr_scheduler.ExponentialLR(optimizer, gamma=0.9)  
</code></pre>
<p>In “get_scheduler(optimizer,opt):” i don’t have access of current epoch but when i use function “def lambda_rule(epoch)” in “scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda_rule)” i have excess of current epoch as below code.</p>
<pre><code class="lang-auto">def lambda_rule(epoch):
   lr_l = 1.0
   return lr_l
scheduler = lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda_rule)

</code></pre>
<p>After every epoch, scheduler.step() is calling. How can i access current epoch is in “get_scheduler”? and what is the reason that i have access of current epoch in lambda_rule(epoch) even i am not passing “epoch” during call.</p>
<p>Second issue is, when “lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda_rule)” called it can return float value 1 but when calling  "lr_scheduler.ExponentialLR(optimizer, gamma=lambda_rule) and return 0.9 it shows below error.</p>
<p>“TypeError: unsupported operand type(s) for *: ‘float’ and ‘function’”</p>
Obtain the attention weights within Transformer class

username: PaulOppelt
cooked: <p>Hey,</p>
<p>i have initialized a transformer-encoder block using:</p>
<p>“encoder_layer = nn.TransformerEncoderLayer(d_model, nhead, dim_feedforward, dropout)”<br>
“transformer = nn.TransformerEncoder(encoder_layer, num_layers)”,</p>
<p>how can I access the attention weights that are returned by “F.multi_head_attention_forward”?<br>
since “need_weights” is manually set to false within the encoder_layer “_sa_block”  function I don’t know how to access them.</p>
<p>best,<br>
Paul</p>

username: AbdulsalamBande
cooked: <aside class="quote no-group" data-username="PaulOppelt" data-post="1" data-topic="165456">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pauloppelt/48/54669_2.png" class="avatar"> PaulOppelt:</div>
<blockquote>
<p>TransformerEncoderLayer</p>
</blockquote>
</aside>
<pre><code class="lang-auto">import torch.nn as nn
encoder_layer = nn.TransformerEncoderLayer(d_model=2, nhead=1,batch_first=True,dropout=0,dim_feedforward=2)
attention_weights = encoder_layer.state_dict()['self_attn.in_proj_weight']
</code></pre>

username: PaulOppelt
cooked: <p>that is only the matrix that contains the projections matrices to produce q,k,v together with the input. The attention matrix I mean it the result of softmax(q @ k.T). I found a different way by inheriting from <code>TransformerEncoderLayer</code> and setting <code>need_weights</code> to true in <code>_sa_block</code>. People from Pytorch told be they may build in the support in a later updates.</p>

username: jasperhyp
cooked: <p>Crap I remember it was True before… I spent so long time debugging and this is why??</p>

username: sam-my-2023
cooked: <p>Well, somehow pretty late, but i met the similar situation, but my lazy intuition tell me to recalculate the atten using encoder_coder.encoder_layer.self_attn( input)[1], I will only consider the first layer’s attn, since the attn after first layer not sure still entagled with the position</p>
GPU is "only" 6x faster than CPU

username: whiplash451
cooked: <p>I run the following code on CPU and GPU.  I observe a disappointing speedup on GPU.</p>
<pre><code class="lang-auto">def build_dataset(words):
    block_size = 3
    X, Y = [], []
    for w in words:
        context = [0] * block_size
        for ch in w + '.':
            ix = stoi[ch]
            X.append(context)
            Y.append(ix)
            context = context[1:] + [ix]

    X = torch.tensor(X).to(device)
    Y = torch.tensor(Y).to(device)
    return X, Y

import random
random.seed(42)
random.shuffle(words)
n1 = int(0.8 * len(words))
n2 = int(0.9 * len(words))

X_train, Y_train = build_dataset(words[:n1])
X_dev, Y_dev = build_dataset(words[n1:n2])
X_test, Y_test = build_dataset(words[n2:])


g = torch.Generator().manual_seed(2147483647)
C = torch.randn((27,10)).to(device)
W1 = torch.randn((30, 200)).to(device)
b1 = torch.randn((1, 200)).to(device)
W2 = torch.randn((200, 27)).to(device)
b2 = torch.randn((1, 27)).to(device)
parameters = [C, W1, b1, W2, b2]
for p in parameters:
    p.requires_grad = True

%%timeit
batch_size = 25000
for i in range(5):
    # minibatch
    ix = torch.randint(0, X_train.shape[0], (batch_size,))

    # forward pass
    emb = C[X[ix]]
    h = torch.tanh(emb.view(-1, 30) @ W1 + b1)
    logits = h @ W2 + b2
    loss = F.cross_entropy(logits, Y_train[ix])
    # backward pass
    for p in parameters:
        p.grad = None
    loss.backward()
    # update
    lr = 0.01
    for p in parameters:
        p.data += -lr * p.grad
    stepi.append(i)
    lossi.append(loss.log10().item())
    print(i, loss)
</code></pre>
<p>I am using an A4000 GPU and 8 vCPUs (AMD EPYC 7502, 2.5MHz).  When I set device to ‘cuda’, the training code runs in 40ms.  When I set the device to ‘cpu’, the same training code runs in 250ms.</p>
<p>I have set to batch_size to maximize GPU usage as per nvidia-smi.  In CPU mode, htop shows that all eights vCPUs are used at full capacity.</p>
<p>I expected the GPU to bring much more speedup.  Am I missing something?</p>

username: ptrblck
cooked: <p>Your GPU workload seems to be tiny besides the large indexing operation, so you might be CPU limited. Profile your use case with the native PyTorch profiler or Nsight Systems to understand the performance and bottlenecks of your code better. If this use case represents your real workload, you might want to check CUDA Graphs to reduce the CPU workload.</p>

username: whiplash451
cooked: <p>Thanks for your help!</p>
MultiheadAttention for cross attention with beam search (generation), redundant computation for keys/values?

username: AlbertZeyer
cooked: <p>E.g. think of an encoder-decoder architecture.</p>
<p>When you do generation, you will frequently have different queries for the same keys and values based on encoder outputs. For training, that can be a single <code>MultiheadAttention.forward</code> call, but for generation, you would call <code>MultiheadAttention.forward</code> for every decoder step. But it looks like this will do the same projection on keys and values over and over in every decoder step? This looks like a lot of redundant computation? Is PyTorch somehow clever to automatically optimize this away? Or is <code>MultiheadAttention.forward</code> not used for generation?</p>
<p>I checked both the <code>torch.nn.MultiheadAttention</code> and also looked at Fairseq <code>MultiheadAttention</code> (<a href="https://github.com/facebookresearch/fairseq/blob/da8fb630880d529ab47e53381c30ddc8ad235216/fairseq/modules/multihead_attention.py#L63" rel="noopener nofollow ugc">here</a>) and they both seem to have this API.</p>
<p>This is very confusing to me. Why would you do this so inefficient? Or do I misunderstand something?</p>
Your module has parameters that were not used in producing loss

username: yupeng_chou
cooked: <p>why the gradient can not backward to self.fc3?<br>
class MatchingMargin(nn.Module):<br>
def <strong>init</strong>(self, s=1, m=0.1, type=‘prod’):<br>
super(MatchingMargin, self).<strong>init</strong>()</p>
<pre><code>    self.type = type
    self.s = nn.Parameter(torch.randn(1,)) if s is None else s
    self.m = m
    self.prototype = nn.Parameter(torch.randn(260, 256))
    self.fc3 = nn.Linear(272,128)

    alphahet = get_alphabet()
    dt_tree = DecomposeTree()

    label_encodings = []
    for char in alphahet:
        char_embedding = dt_tree.query_feature(char)
        label_encodings.append(char_embedding)

    # label_encodings = label_encodings[:template.size(0)]

    self.label_encodings = torch.from_numpy(
        np.stack(label_encodings, 0),
    ).float()

    # print(self.label_encodings.requires_grad)
    self.label_encodings = nn.Parameter(self.label_encodings)

    # self.register_buffer('W', self.label_encodings)

def forward(self, input, template, label=None,train=True):
    device = input.device
    # self.W = self.W[:template.size(0)].to(device)
    label_encodings = self.label_encodings[:template.size(0)].to(device)
    tree_vec = self.fc3(label_encodings)

    if self.type == 'prod':
        cosine = F.linear(input, template)
        dis = F.linear(input,tree_vec)
        cosine = cosine + dis
    elif self.type == 'cosine':
        cosine = F.linear(F.normalize(input), F.normalize(template))
    elif self.type == 'euclid':
        cosine = torch.norm(input.unsqueeze(1) - template.unsqueeze(0), p=2, dim=2)
        # print('input', input.size(), 'template', template.size(), 'cosine', cosine.size())
    else:
        raise NotImplementedError()

    if label is not None and np.fabs(self.m) &gt; 1e-4:
        one_hot = torch.zeros_like(cosine)
        one_hot.scatter_(1, label.view(-1, 1), 1.0)
        cosine = (cosine - one_hot * self.m)

    output = cosine * self.s
    return output
</code></pre>

username: ptrblck
cooked: <p>Are the parameters in <code>self.fc3</code> not receiving any gradients or is this layer not back propagating them? In the latter case I would guess the <code>label_encodings</code> might not be differentiable.</p>
Are there profiling tools to measure LibTorch (C++) JIT overhead vs Torch library runtime?

username: yetanadur
cooked: <p>We convert PyTorch to TorchScript IR and use libtorch run time (cpu &amp; gpu). We would like to profile this application to find out how much execution time is spent in JIT vs Torch libraries. Are there any recommended profiling tools/methods ?</p>
<p>Cheers,<br>
Yetanadur</p>

username: CaRoLZhangxy
cooked: <p>I’ve encountered a similar issue. Have you happened to find a good solution?</p>
Dataset Bias Concerns in Mask RCNN

username: H_Jahan
cooked: <p>Hello<br>
I’m preparing a dataset for a Mask R-CNN model, involving images of cats and smaller, distinct spots on these cats. While the dataset has more instances of “spots” than “cats,” the latter covers a much larger area in the images. I’m concerned this might bias the model toward the “cat” class due to its larger pixel coverage.</p>
<p>My question is:</p>
<p>Could this difference in area coverage introduce significant training bias towards the “cat” class?</p>
<p>Any advice or resources would be appreciated.</p>
What are the advantages of using Python over a language like C++ for machine learning?

username: gracylayla
cooked: <p>I recently started studying deep learning and other ML techniques, and I started searching for frameworks that simplify the process of build a net and training it, then I found <a href="https://mindmajix.com/tensorflow-training" rel="nofollow noopener">TensorFlow</a>, having little experience in the field, for me, it seems that speed is a big factor for making a big ML system even more if working with deep learning, so why python was chosen by Google to make TensorFlow? Wouldn’t it be better to make it over an language that can be compiled and not interpreted?</p>
<p>What are the advantages of using Python over a language like C++ for machine learning?</p>

username: ptrblck
cooked: <p>I cannot comment of Tensorflow and I think their board or stackoverflow would be a better place to dig into this framework.</p>
<p>However, PyTorch uses C/C++/CUDA as the underlying functions.<br>
Python is quite popular in the scientific community as it’s really easy to create your scripts using this language.<br>
The heavy workload is performed by highly optimized code, so that you usually don’t have to worry about it.</p>
<p>Some core devs can give you definitely more insight on this topic.<br>
Also. have a look at <a href="https://openreview.net/pdf?id=BJJsrmfCZ">their paper</a>.</p>

username: chenglu
cooked: <p>Python is</p>
<ul>
<li>Easy to learn</li>
<li>More like a natural language</li>
<li>Code is very straightforward. You can look at just one python file and know where everything comes from. (unless the coder using <code>import *</code>)</li>
<li>…</li>
</ul>
About batched effect in backpropagation of matmul

username: Ther-nullptr
cooked: <p>How does bp work in batched matmul scene? assume we have a linear, the shape of x is <code>[b,m,k]</code>, shape of W is <code>[k,n]</code>:</p>
<p>$$<br>
y = xW<br>
$$</p>
<p>then:</p>
<p>$$<br>
\frac{\partial L}{\partial W} = x^T\frac{\partial L}{\partial y}<br>
$$</p>
<p>so what had happened in backward of W?</p>
<ol>
<li>We save the full $x^T$ tensor <code>[b,k,m]</code>, and execute the batched matmul <code>[b,k,m] x [b,m,n]</code>, finally reduce the result to <code>[k,n]</code> to get the gradient.</li>
<li>We directly save the averaged $x^T$ tensor <code>[k,m]</code>, and execute the batched matmul <code>[k,m] x [m,n]</code> to get the gradient.</li>
</ol>

username: soulitzer
cooked: <p>We do (1), the second one doesn’t compute the same quantity. Would be cool to reduce ahead of time to save memory though lol.</p>
Tensor.cuda() takes ~2 seconds

username: krebso
cooked: <p>Hello there!</p>
<p>I am using captum to run some explanations of my model (vgg16). When using occlusion and setting pertubations_per_eval=100 (allows to run 100 forward passes simultaneously), it needs ~23GB of GPU. In the next (and every subsequent) iteration of my loop, calling tensor.cuda() on input takes ~2 seconds. Is it because of the GPU context switching? Is there something I can do about it?</p>
<p>I am using batch_size=1 and the GPU has ~46GB.</p>
<p>I am happy to provide any code samples, if needed.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>How are you measuring the time? To properly profile the GPU execution time you would need to synchronize the GPU. Otherwise ops might be accumulating runtimes.</p>

username: krebso
cooked: <p>Hey!</p>
<p>Initially I was just printing the time difference of the two calls, I did not realize that the call is not blocking, if the computation happens on GPU. I looked it up, and this should do the job:</p>
<pre data-code-wrap="python"><code class="lang-python">for t, _, _ in dataloader:
    c = time.time()
    t = t.cuda()
    torch.cuda.synchronize()
    print(time.time() - c)
    a = time.time()
    occ.attribute(tensor, ..., pertubations_per_eval=100)
    torch.cuda.synchronize()
    print(time.time() - a)
    ...
</code></pre>
<p>Now the values I am getting are:<br>
<code>t.cuda</code> → 0.000 seconds - As expected, since it is one (3, 512, 512) tensor<br>
<code>occ.attribute</code> → ~1.98 seconds - Here comes my confusion, I would expect setting the value to 100 (instead of original 1, where it took ~2.25 seconds) would make it run much faster, but it did not.</p>
<p>Therefore I am curious whether the lack of speed-up is something related to the GPU having to free and allocate a lot of memory, or something happening in Captum (maybe generating the occlusion patches, haven’t looked into it yet).</p>

username: ptrblck
cooked: <p>PyTorch caches the memory and does not repeatedly allocate and free it as it would result in a terrible performance. I’m not familiar enough with Captum and don’t know what this method does, but you could profile it’s internals.</p>
Creating a dataset that returns two images efficiently

username: madkr
cooked: <p>I am working on a project where I have to simultaneously load an image along with the features of the image. I have pre-computed the features (single channel, same dimensions as the image) and they’re present on the same disk with the same name and structure as the source files.<br>
The idea is to load the images simultaneously, perform the same preprocessing on both (rotation/cropping/affine), and normalize only the source images.<br>
I have tried to build a dataset, but it is way too slow and often blocks the dataloader. The code is something like this:</p>
<pre><code class="lang-auto">class customdataset(torchvision.datasets.ImageFolder):
    def __init__(self,
                 src,
                 feat_src):
        super().__init__(src)
        self.src = src
        self.feat_src= feat_src
        
        self.transformation = [transformation,
                                          normalization]
    def __getitem__(self, idx):
        img_path, label = self.imgs[idx]
        feat_path = img_path.replace(self.src, self.feat_src)
        
        # open imgs
        img = self.loader(img_path)
        feat= self.loader(feat_path).convert("L")
        
        img = # convert image to tensor
        feat= # convert features to tensor
        
        combo_img = torch.cat([img, feat], axis = 0)   # concatenate images 
        combo_img = self.transforms(combo_img )    # apply transformation
            
        img = combo_img [:-1]     # isolate images
        feat= combo_img [-1]       # isolate features
    
        
        return img, feat, label
</code></pre>
<p>Right now, it’s blocking a lot with anything other than a PCIe NVMe 4.0 SSD (tried on cloud and remote servers with beefy GPU and CPU configs, barely could do 100 steps of the dataloader in 1 minute). I know that this can be much faster, as single-image loops using ImageFolder run at ~250 dataloader steps in 1 min.</p>
<p>I have tried a lot of things to no avail. I want this code to be as efficient as possible as the training loop will run for ~2-4 days. Could anyone please suggest how I can make it more efficient? Thanks in advance.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="madkr" data-post="1" data-topic="193837">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/madkr/48/66312_2.png" class="avatar"> madkr:</div>
<blockquote>
<p>Right now, it’s blocking a lot with anything other than a PCIe NVMe 4.0 SSD</p>
</blockquote>
</aside>
<p>I don’t know which setup you are using currently but if the data loading bandwidth is the current bottleneck your options might be limited to speed up the code besides obviously upgrading your storage. You could try to increase the number of workers hopefully allowing them to prefetch the data fast enough.</p>