akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
TypeError: 'module' object is not callable for loop

Cori_ander: <p>Hello,<br>
I’m a beginner of pytorch.<br>
I’m trying to classicification of VGG using CIFAR10.<br>
The last of course, I’m making the train loop as belows,<br>
But I got the typeerror but I don’t what is wrong.<br>
Please help me.</p>
<p>----(code)</p>
<p>for epoch in range(100):<br>
for data, label in train_loader:<br>
optim.zero_grad()<br>
preds = model(data.to(device))</p>
<pre><code>  loss = nn.CrossEntropyLoss()(grads, label.to(device))
  loss.backward()
  optim.step()
</code></pre>
<p>if epoch==0 or epoch%10==9:<br>
print(f"epoch{epoch+1} loss:{loss.item()}")</p>
<p>---------(error)</p>
<p>TypeError                                 Traceback (most recent call last)<br>
 in <br>
7<br>
8 for epoch in range(100):<br>
----&gt; 9   for data, label in train_loader:<br>
10       optim.zero_grad()<br>
11       preds = model(data.to(device))</p>
<p>4 frames<br>
/usr/local/lib/python3.8/dist-packages/torchvision/datasets/cifar.py in <strong>getitem</strong>(self, index)<br>
116<br>
117         if self.transform is not None:<br>
 → 118             img = self.transform(img)<br>
119<br>
120         if self.target_transform is not None:</p>
<p>TypeError: ‘module’ object is not callable</p>

ptrblck: <p>Could you post the dataset definition and in particular what you’ve passed as the <code>transform</code> object to it?</p>

Cori_ander: <p>Of course!<br>
Here’s my code for datasets!</p>
<p>------------(code)</p>
<p>import torch<br>
import torchvision<br>
from torchvision import datasets, models, transforms</p>
<p>training_data = torchvision.datasets.CIFAR10(root=‘./data’, train=True,<br>
download=True, transform=transforms)<br>
test_data = torchvision.datasets.CIFAR10(root=‘./data’, train=False,<br>
download=True, transform=transforms)</p>
<p>train_loader = DataLoader(training_data, batch_size=32, shuffle=True)</p>
<p>test_loader = DataLoader(test_data, batch_size=32, shuffle=False)</p>

ptrblck: <p>You are passing the <code>transforms</code> module directly, which is causing the issue.<br>
Create actual transformation objects and it should work, e.g.</p>
<pre><code class="lang-python">training_data = torchvision.datasets.CIFAR10(root='./data', train=True,
                                             download=False, transform=transforms.ToTensor())
</code></pre>
<p>EDIT: Take a look at <a href="https://pytorch.org/vision/stable/transforms.html">these docs</a> for more information how transformations are applied.</p>

Cori_ander: <p>Thank you! I solved this problem.</p>
Why no fold and unfold for 5d and 3d tensors?

shengchunnan: <p>We know that any convolution operations could be divided into three operations: unfold → matmul --&gt;fold, does not matter this convolution is conv1d or conv2d or conv3d.<br>
However, existing fold() and unfold() APIs allow 4D tensors only.<br>
Maybe I can implement conv3d and conv1d equivalent fold() and unfold() operations of my own using openBLAS, CUDA or cuBLAS … But I believe pytorch implementations would be the most efficient and qualified.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-implement-a-convolutional-layer/68211/7">This post</a> uses the more flexible <code>tensor.unfold</code> method to compute the convolution and you could use it as a template for a 1d or 3d conv layer.</p>
Getting TypeError: 'int' object is not callable on creating dataset

granth_jain: <p>I am trying to train a timeseries model with X_train shape =(-1, length, features), y_train shape=(-1)<br>
But I am getting error on creating the dataset out of it. Below is my code</p>
<blockquote>
<pre><code>X_train = load(open(r"X_train_"+str(dataset_idx)+"_"+str(model_idx)+".pkl", 'rb'))
y_train = load(open(r"y_train_"+str(dataset_idx)+"_"+str(model_idx)+".pkl", 'rb'))

print(X_train.shape, y_train.shape)

dataset_train = torch.utils.data.TensorDataset(X_train, y_train)
</code></pre>
</blockquote>
<p>Below is output:</p>
<blockquote>
<p>(3654, 20, 4) (3654,)</p>
<p>Traceback (most recent call last):</p>
<p>File ~\anaconda3\lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec<br>
exec(code, globals, locals)</p>
<p>File d:\scripts\pytorch.py:206<br>
main(model_idx, dataset_idx)</p>
<p>File d:\scripts\pytorch.py:176 in main<br>
dataset_train = torch.utils.data.TensorDataset(X_train, y_train)</p>
<p>File ~\anaconda3\lib\site-packages\torch\utils\data\dataset.py:189 in <strong>init</strong><br>
assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors), “Size mismatch between tensors”</p>
<p>File ~\anaconda3\lib\site-packages\torch\utils\data\dataset.py:189 in <br>
assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors), “Size mismatch between tensors”</p>
<p>TypeError: ‘int’ object is not callable</p>
</blockquote>
<p>Can someone please let me know what I am missing here.<br>
Thanks</p>

ptrblck: <p>I guess you are passing numpy arrays to your <code>TensorDataset</code> instead of tensors:</p>
<pre><code class="lang-python">a = np.random.randn(10)
a.size(0)
# TypeError: 'int' object is not callable
a.size
# 10

x = torch.randn(10)
x.size(0)
# 10
</code></pre>
<p>which will then raise the error since <code>size</code> is an attribute for numpy arrays and a method for tensors.</p>
Huge lag time between epochs

omarmetrx: <p>Hello,</p>
<p>I am running a training with <code>torchrun  --nnodes 1  --nproc_per_node 2  training.py</code>. Each epoch step do parallalize and proceed in shorter time as expected. However, there is a delay about 20 mins before proceeding to the next epoch!</p>
<p>When running on single gpu (without torchrun), I don’t get this delay.</p>
<p>Any ideas to solve this is appreciated.</p>
<p>Thanks in advance!</p>

agu: <p>Providing targeted help is difficult without a way to reproduce your issue <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>
<p>Do you have a minimal script? Could you provide some hardware information?</p>

omarmetrx: <p>Hi <a class="mention" href="/u/agu">@agu</a>.</p>
<p>I am using this script here <a href="https://github.com/Project-MONAI/tutorials/blob/main/pathology/hovernet/training.py" class="inline-onebox" rel="noopener nofollow ugc">tutorials/training.py at main · Project-MONAI/tutorials · GitHub</a>.</p>
<p>It is basically a pytorch ignite wrapper.</p>
<p>For the hardware, I am using 2 A100 gpus.</p>
<p>What else I can provide?</p>

J_Johnson: <aside class="quote no-group" data-username="omarmetrx" data-post="3" data-topic="174176">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omarmetrx/48/56268_2.png" class="avatar"> omarmetrx:</div>
<blockquote>
<p>I am using this script here <a href="https://github.com/Project-MONAI/tutorials/blob/main/pathology/hovernet/training.py" rel="noopener nofollow ugc">tutorials/training.py at main · Project-MONAI/tutorials · GitHub </a>.</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/omarmetrx">@omarmetrx</a><br>
If you’re using some compilation of code written partially in PyTorch, you might try raising an issue on their GitHub page as they will be most familiar with their library.</p>
<p>One thing you might check is if you’re using all of the same versions of libraries they used when they developed the project, including CuDNN and CUDA.</p>
[W NNPACK.ccp:53] Could not initialize NNPACK! Reason: Unsupported hardware

exihs: <p>This warning does not prevent the program from running, however it does cause python to quit unexpectedly after a few seconds, which I believe is linked to this error.</p>
<p>torch==1.13.1<br>
torchvision==0.14.1</p>
<p>Platform: MacOS (M1 chip)<br>
My program uses easyocr, which depends on both torch and torchvision.</p>
<p>I have tried uninstalling both torch and torchvision via <code>pip3 uninstall torch torchvision</code>, and building both from source with NNpack disabled for torch</p>
<p>When building torch:<br>
<code>USE_NNPACK=0 python3 setup.py develop</code></p>
<p>When building torchvision:<br>
<code>python3 setup.py develop</code></p>
<p>Besides the NNPack warning message in the terminal, here is details of when python crashes<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/1/1101a31881fa9c8790f5ad1d5be8b74ec94be299.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1101a31881fa9c8790f5ad1d5be8b74ec94be299" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/1101a31881fa9c8790f5ad1d5be8b74ec94be299_2_690x377.jpeg" alt="image" data-base62-sha1="2qrCIoxdk4nvsvMvx0LDcFATt0Z" width="690" height="377" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/1101a31881fa9c8790f5ad1d5be8b74ec94be299_2_690x377.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/1101a31881fa9c8790f5ad1d5be8b74ec94be299_2_1035x565.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/1/1101a31881fa9c8790f5ad1d5be8b74ec94be299_2_1380x754.jpeg 2x" data-dominant-color="292929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1758×962 81.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How to train same model with and without dataparallel

ronzhou: <p>Hi. I found a situation that training a module with or without dataparallel with return a different module.</p>
<p>The trainig process is simple:</p>
<pre><code class="lang-python">model = Model(input_size, output_size)
model = model.cuda()

if no dp:
    pass
elif 2 gpu:
  model = nn.DataParallel(model, device_ids=[0,1])
elif 3 gpu:
  model = nn.DataParallel(model, device_ids=[0,1,2])


for data in rand_loader:
    input = data.to('cuda')
    output = model(input)

    loss = compute_loss()
    loss.backwards()

</code></pre>
<p>The results are different when training with one gpu (no dataparallel), or training with different size of device_ids, which means result different with 2 or 3 gpu.</p>
<p>What should I do if I want exactly same module in those three scenarios? Any help will be really appreciated!</p>

mrshenli: <p>Hey <a class="mention" href="/u/ronzhou">@ronzhou</a>, could you please clarify what you mean by “results are different”? Are there huge gaps in the loss curve?</p>

ronzhou: <p>You can reproduce the problem with the following code.</p>
<pre data-code-wrap="import"><code class="lang-plaintext">from torch import nn
torch.manual_seed(42)
layer = nn.TransformerEncoderLayer(d_model=20, nhead=5, batch_first=True)
model = nn.TransformerEncoder(layer, num_layers=5)
model.to('cuda:2')
model_parallel = nn.DataParallel(model, device_ids=[2,3])
model_parallel2 = nn.DataParallel(model, device_ids=[2,5,6])
input_ = torch.randn(size=(31, 14, 20)).to('cuda:2')
model_result = model(input_)
model_parallel_result = model_parallel(input_)
model_parallel_result2 = model_parallel2(input_)
abs_max = lambda a,b : f"{(a - b).abs().max().item():.3f}"
print(abs_max(model_parallel_result, model_result)) #1.576
print(abs_max(model_parallel_result, model_parallel_result2)) #1.416
</code></pre>
<p>If the model was in the eval mode, the results are identical. But the issue is that I have to train the model using dataparallel. I think the difference should come from the dropout.</p>
Problems with batchsize when using DataParallel and DistributedDataParallel

SCN-THU: <p>Suppose I have 4 GPUs, I use DataParallel and DistributedDataParallel respectively to train the model, and pass batch_size=64 in DataLoader. So what is the equivalent actual batchsize (when using a single GPU)?</p>

agu: <p>For <code>DistributedDataParallel</code>, you should use the <code>DistributedSampler</code> in combination with the <code>DataLoader</code>, or else each rank will process the same data, which will be equivalent to a single GPU processing a dataset that has the original dataset replicated 4 times.</p>
<p>I am not too familiar with <code>DataParallel</code>, and we recommend trying <code>DistributedDataParallel</code> if possible. If your question is only for conceptual understanding, I apologize that I do not know <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

SCN-THU: <p>Thank you for your help <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
Understanding bottlenecks in training loop speed

thebeginning_isnigh: <p>I’m trying to understand something about loops in PyTorch. I timed this function (modified from docs):</p>
<pre><code class="lang-auto">    for batch, (X, y) in enumerate(dataloader):
        print("Shape: ", X.shape)
        X, y = X.to(device), y.to(device)
        
        for i in range(10000):
            # Compute prediction error
            pred = model(X)
            loss = loss_fn(pred, y)

            # Backpropagation
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

        break
</code></pre>
<p>This function will run training on a single batch 10,000 times (I understand this is pointless as it would just memorize that specific batch).</p>
<p>The reason I wrote it is to understand why all of the following are true:</p>
<ul>
<li>This ran for a really small network ([784 in] → [512 dense] → ReLU → [10 out])</li>
<li>I ran this on a 4090. It’s a 24gb card with 16k cuda cores and 2.3ghz clock speed.</li>
<li>The time it takes to complete this is about 3.6 seconds.</li>
</ul>
<p>From what I understand about deep learning on GPUs, 3.6 seconds seems way too high.</p>
<p>If the GPU has plenty of memory to keep the entire model (along with state for the batch and gradients), 10,000 iterations for a 3 layer network should complete well under a few hundred thousand clock cycles. (This could also be a complete misunderstanding of how this works, please feel free to correct me if it’s wrong)</p>
<p>My best theory for why:<br>
The for loop runs on the CPU, so each iteration, the GPU communicates back to the CPU that 1 iteration is complete, and the for loop continues. But even that communication time should be marginal and shouldn’t cause such a slow training time?</p>
<p>I can’t find any answers on a) whether it should be faster, and b) why it isn’t.</p>
<p>I’d appreciate any insights. Thanks.</p>

ptrblck: <p>CUDA operations are executed asynchronously and since your code is mussing synchronizations it will profile the dispatching and kernel launch overheads at best.<br>
You are right that the CPU is used to schedule the work and especially if your actual GPU workload is tiny these overheads might be visible as “whitespaces” in a visual profile created by the native PyTorch profiler or e.g. Nsight Systems.<br>
In these cases your workload is CPU-limited and the GPU is “starving”.<br>
If these tiny models represent real workloads you could check CUDA graphs, which would eliminate the kernel launch overheads as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">here</a>.</p>

J_Johnson: <p>Try increasing your batch size instead of running 10,000 iterations.</p>
<p>You can also make use of half precision or <a href="https://pytorch.org/blog/what-every-user-should-know-about-mixed-precision-training-in-pytorch/" rel="noopener nofollow ugc">mixed precision</a> for an additional speed up and larger batches.</p>
Layer-wise learning rate in fsdp

tree33: <p>Hi, I’m trying to set different lr for different layers in fsdp models.</p>
<p>For DDP, the <code>model.named_parameters()</code> function gives me all the names and their corresponding weights of all layers. And I can simply filter out the norm layers and apply no weight decay for these layers.</p>
<p>While for fsdp, <code>named_parameters()</code>  gives only <code>._fsdp_wrapped_module.flat_param</code> and a flattened array. I’m wondering if it is possible to also apply no weight decay for norm layers inside fsdp.</p>

agu: <p>You can try out passing <code>use_orig_params=True</code> to the FSDP constructor <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Then, the original parameters will be returned from <code>named_parameters()</code>. Note though, the original parameters’ data will be 1D and only contain what is in the local rank’s shard. Still, you should be able to apply separate learning rates (or other hyperparameters) for each original parameter.</p>