akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Slow data loading

chris: <p>My data loader is very slow. I have around 6700 .pt files of around 80 kb each and when using my custom dataset, the data loader takes a long time, around 30 mins per epoch. It is very quick if i use a limited dataset, i.e. 1000 .pt files. Increasing number of workers allow for faster iteration, but it takes a long time to initialise each workers after each batch. As i work with graphs i use the Pytorch Geometric dataloader, but it should not be much different to the Pytorch loader. Should I save my graphs in another format instead of .pt files? Not sure I understand why this is happening. Here is the code:</p>
<pre><code class="lang-python">class CDataset(Dataset):
  def __init__(self, root, pre_filter=None, pre_transform=None):
      super(CDataset, self).__init__(root, pre_filter, pre_transform)
      
  def atoi(self, text):
    return int(text) if text.isdigit() else text
  
  def natural_keys(self, text):
    return [ self.atoi(c) for c in re.split(r'(\d+)', text) ]
  
  @property
  def raw_file_names(self):
    path_to_raw = os.listdir(self.root+"/raw")
    path_to_raw.sort(key=self.natural_keys)
    return path_to_raw
  
  @property
  def processed_file_names(self):
    names = []
    for i in range(len(self.raw_paths)):
      names.append('data_{}.pt'.format(i))
    names.sort(key=self.natural_keys)
    return names

  def download(self):
    pass

  def process(self):
      i = 0
      for raw_path in self.raw_paths:
        
        data = torch.load(raw_path)
        data = data if self.pre_filter is None else self.pre_filter(data)
        data = data if self.pre_transform is None else self.pre_transform(data)
        torch.save(data, osp.join(self.processed_dir, 'data_{}.pt'.format(i)))
        i += 1

  def len(self):
    return len(self.processed_file_names)

  def get(self, idx):
    data = torch.load(osp.join(self.processed_dir, 'data_{}.pt'.format(idx)))
    return data
</code></pre>

Mohit_Kumar_Pandey: <p>Were you able to figure this out?</p>

Animesh_Basak_Chowdh: <p>Would be great if we get some pointers on this. Dataloader for pytorch geometric has been a real issue.</p>

Deceptrax123: <p>Heyy, Once your .pt files of the graph data objects have been saved. You  dont have to override process(). Remove the process function and include the processed paths property  with the absolute paths of all your .pt files just as you  used raw_paths property. Hope this helps: )</p>
What is error in this code pytorch

AishaMohamed2022: <p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim</p>
<p>train_loader = torch.utils.data.DataLoader(cifar2, batch_size=64,<br>
shuffle=True)</p>
<p>model = nn.Sequential(<br>
nn.Linear(3072, 128),<br>
nn.Tanh(),<br>
nn.Linear(128, 2),<br>
nn.LogSoftmax(dim=1))</p>
<p>learning_rate = 1e-2</p>
<p>optimizer = optim.SGD(model.parameters(), lr=learning_rate)</p>
<p>loss_fn = nn.NLLLoss()</p>
<p>n_epochs = 100</p>
<p>for epoch in range(n_epochs):<br>
for imgs, labels in train_loader:<br>
outputs = model(imgs.view(imgs.shape[0], -1))<br>
loss = loss_fn(outputs, labels)</p>
<pre><code>    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print("Epoch: %d, Loss: %f" % (epoch, float(loss)))
</code></pre>
<p>i have this error<br>
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.Image.Image’&gt;<br>
i can not fixed it?</p>

ptrblck: <aside class="quote no-group" data-username="AishaMohamed2022" data-post="1" data-topic="192972">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aishamohamed2022/48/65556_2.png" class="avatar"> AishaMohamed2022:</div>
<blockquote>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.Image.Image’&gt;</p>
</blockquote>
</aside>
<p>Transform the <code>PIL.Image</code>s to tensors. I don’t know how <code>cifar2</code> is defined, but assume you can pass <code>transform=transforms.ToTensor()</code> to its init.</p>

AishaMohamed2022: <p>it correct ?<br>
imgs,labels=cifar2[0]<br>
to_tensor=transforms.ToTensor()<br>
imge=to_tensor(imgs)<br>
imge.shape</p>

shalom_p: <p>I think <a class="mention" href="/u/ptrblck">@ptrblck</a>  had asked you to pass this transforms to the dataset set class when you are initializing it, there should be a parameter taking transforms if not that then what you have done will also work.</p>

ptrblck: <p>Yes, <a class="mention" href="/u/shalom_p">@shalom_p</a> is right and if you are using <a href="https://pytorch.org/vision/main/generated/torchvision.datasets.CIFAR10.html#torchvision.datasets.CIFAR10"><code>torchvision.datasets.CIFAR10</code></a>, just it to the <code>transform</code> argument. If you are using another class definition, check its docs.</p>
RuntimeError in PyTorch Model with Multi-Class Classification

favoruu39: <p>Hello,</p>
<p>I’m currently working on a PyTorch model for classifying 9 different classes using images of size 722 x 722. However, I keep encountering the following runtime error:</p>
<p><code>RuntimeError: The size of tensor a (32) must match the size of tensor b (9) at non-singleton dimension 1</code></p>
<p>I’m not sure how to resolve this issue. Here’s a brief overview of my model’s structure:</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from making_dataset import *

class Net(nn.Module):
    def __init__(self):

        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1,6,5)
        self.pool = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(6,16,5)

        self.fc1=nn.Linear(16*53*53,120)
        self.fc2=nn.Linear(120,84)
        self.fc3=nn.Linear(84,9)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        print(x.shape)
        x = x.view(-1, 16 * 53 * 53)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


net = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

train_loader, valid_loader, test_loader = get_dataloader(32,0.7, 0.15)


for epoch in range(2):  

    net.train()  
    for i, data in enumerate(train_loader, 0):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()


    net.eval()  
    valid_loss = 0.0
    correct = 0
    total = 0
    with torch.no_grad():  
        for data in valid_loader:
            inputs, labels = data
            outputs = net(inputs)
            loss = criterion(outputs, labels)
            valid_loss += loss.item()
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()


    print(f'Epoch {epoch+1}, Train loss: {loss:.3f}, Valid loss: {valid_loss/len(valid_loader):.3f}, Accuracy: {100 * correct / total:.2f}%')

print('Finished Training')
</code></pre>

ptrblck: <p>Replace <code>x = x.view(-1, 16 * 53 * 53)</code> with <code>x = x.view(x.size(0), -1)</code> as the former code might change the batch size.</p>
Normalizing 3D volumes

887574002: <p>Hi, I have a silly question. Is the following class the right normalization for 3D grayscale images?</p>
<pre><code class="lang-auto">    def __init__(self):  
        pass
    def __call__(self,vol):
        vol = vol-vol.mean()/vol.std()
        return(vol) ```

If so, should I expect after applying this nor,alization the mean of transformed image ill be zero and std will be one?</code></pre>

ptrblck: <p>You should wrap the subtraction into parentheses, otherwise you would scale the <code>mean</code> value by the inverse <code>std</code>.<br>
If that’s fixed, your output should have a zero mean (or something around <code>1e-6</code>) and a unit variance.</p>

887574002: <p>It was solved. Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a>. The variance also is near 1, not exactly 1, is that true?</p>

ptrblck: <p>How large is the difference between the calculated variance and the theoretical <code>1.</code>?</p>

887574002: <p>This is mean <code>1.3271347e-07</code> and this is std <code>0.9999998</code>.</p>

ptrblck: <p>This seems to be alright, as the difference is in the floating point precision limits.</p>

887574002: <p>Thank you for your answer.</p>

ado_sar: <p>Correct me if I am wrong, but he could also use <code>normalize</code> from <code>torchvision.transforms.functional</code>.</p>
<pre><code class="lang-auto">from torchvision.transforms.functional import normalize
normalize(x, mean, std)
</code></pre>
<p>Although the <code>normalize</code> function is used for 2D images, by looking on the source code, if you pass just scalars, then it is equivalent to:</p>
<pre><code class="lang-auto">(x - mean) / std
</code></pre>

ptrblck: <p>Yes, it should work today, but note that the original question is from July 2020 while transformations on tensors were introduced in <code>torchvision==0.8.0</code> in Dec. 2020 as seen in the <a href="https://github.com/pytorch/vision/releases/tag/v0.8.0">release notes</a>.</p>

zeynepdeniz: <p>hi, I am trying to normalize my 3d grayscale image dataset. Should I use the mean and std values of the dataset or the particular image? If I try to normalize with the mean and std values of the whole dataset not every image has zero mean and one std so I am confused.</p>
Error in importing pre-trained model

Taeha_Jeong: <p>Hello, I tried to load pre-trained <code>efficientnet_b0</code> using the code found <a href="https://pytorch.org/vision/stable/models.html" rel="noopener nofollow ugc">here</a>.</p>
<p>I used the following:</p>
<pre data-code-wrap="python"><code class="lang-python">from torchvision.models import efficientnet_b0, EfficientNet_B0_Weights

weight = EfficientNet_B0_Weights.DEFAULT
effModel = efficientnet_b0(weights=weight)
</code></pre>
<p>but I keep getting the error message saying:</p>
<pre data-code-wrap="python"><code class="lang-python">Traceback (most recent call last):
  File "path", line 13, in &lt;module&gt;
    effModel = efficientnet_b0(weights=weight)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\Python\Python311\Lib\site-packages\torchvision\models\_utils.py", line 142, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "path\Python\Python311\Lib\site-packages\torchvision\models\_utils.py", line 228, in inner_wrapper
    return builder(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\Python\Python311\Lib\site-packages\torchvision\models\efficientnet.py", line 770, in efficientnet_b0
    return _efficientnet(
           ^^^^^^^^^^^^^^
  File "path\Programs\Python\Python311\Lib\site-packages\torchvision\models\efficientnet.py", line 360, in _efficientnet
    model.load_state_dict(weights.get_state_dict(progress=progress, check_hash=True))
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\Python\Python311\Lib\site-packages\torchvision\models\_api.py", line 90, in get_state_dict
    return load_state_dict_from_url(self.url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\Python\Python311\Lib\site-packages\torch\hub.py", line 766, in load_state_dict_from_url
    download_url_to_file(url, cached_file, hash_prefix, progress=progress)
  File "path\Python\Python311\Lib\site-packages\torch\hub.py", line 663, in download_url_to_file
    raise RuntimeError(f'invalid hash value (expected "{hash_prefix}", got "{digest}")')
RuntimeError: invalid hash value (expected "3dd342df", got "7f5810bc96def8f7552d5b7e68d53c4786f81167d28291b21c0d90e1fca14934")
</code></pre>
<p>Could someone take a look at this please? Thank you a lot!</p>

fibikong: <p>I cannot reproduce that error.<br>
I think you should check the version of python or pytorch.</p>
How can I read hdf5 files stored as 1-D array. and view them as images?

Adex: <p>I have a large image classification dataset stored in the format <code>.hdf5</code>. The dataset has the labels and the images stored in the <code>.hdf5</code> file. I am unable to view the images as they are store in form of an array. The dataset reading code that I have used is as follows,</p>
<pre><code class="lang-auto">    import h5py
    import numpy
    f = h5py.File('data/images.hdf5', 'r')
    print(list(f.keys()))
</code></pre>
<p><code>['datasets']</code></p>
<pre><code class="lang-auto">    group = f['datasets']
    list(group.keys())
</code></pre>
<p><code>['car']</code></p>
<p>Now when I read the group <code>cars</code> I have the following output,</p>
<pre><code class="lang-auto">    data = group['car']
    data.shape,data[0].shape,data[1].shape
</code></pre>
<p><code>((51,), (383275,), (257120,)</code></p>
<p>So it looks like there are <code>51</code> images for label <code>car</code> and images are stored as <code>383275</code> and <code>257120</code> dimensional arrays, with no information about their height and width dimensions. I want to save the images as RGB again.<br>
Next following the code <a href="https://stackoverflow.com/questions/52394252/convert-h5-file-to-jpg-with-python" rel="noopener nofollow ugc">here</a>, I tried to read the images.</p>
<pre><code class="lang-auto">    import numpy as np
    from PIL import Image
    
    # hdf = h5py.File("Sample.h5",'r')
    array = data[0]
    img = Image.fromarray(array.astype('uint8'), 'RGB')
    img.save("yourimage.thumbnail", "JPEG")
    img.show()
</code></pre>
<p>Unfortunately, the following error is received.</p>
<pre><code class="lang-auto">    File /usr/local/lib/python3.8/dist-packages/PIL/Image.py:784, in Image.frombytes(self, data, decoder_name, *args)
        781 s = d.decode(data)
        783 if s[0] &gt;= 0:
    --&gt; 784     raise ValueError("not enough image data")
        785 if s[1] != 0:
        786     raise ValueError("cannot decode image data")
    
    ValueError: not enough image data
</code></pre>
<p>References I have already checked the <a href="https://discuss.pytorch.org/t/save-torch-tensors-as-hdf5/39556">pytorch discuss forums questions like these</a>, <a href="https://forum.hdfgroup.org/t/how-to-store-image-in-hdf5-files-with-h5py/8892" rel="noopener nofollow ugc">hdf group help library</a> etc.<br>
Any help will be highly useful. Thanks.</p>

dfarmer: <p>Without access to the file it’s pretty hard to say, but as a shot in the dark – try seeing if whoever made the file attached the size information in dataset’s attributes (i.e., <code>data.attrs</code> ). Assuming that works out then you’ll want to <code>numpy.reshape</code> the 1D array into n x w x h and then convert those to images.</p>
How to compute accuracy when using v2.Mixup?

Fatemeh_A: <p>Hi all,</p>
<p>In the context of <a href="https://pytorch.org/vision/main/auto_examples/transforms/plot_cutmix_mixup.html#sphx-glr-auto-examples-transforms-plot-cutmix-mixup-py" rel="noopener nofollow ugc">v2.Mixup</a>, how would one go about calculating accuracy when ‘lambda,’ target_a, and target_b are not returned?</p>
<p><a href="https://github.com/facebookresearch/mixup-cifar10/blob/main/train.py#L161" rel="noopener nofollow ugc">The original implementation</a> involves accuracy calculation by multiplying with ‘lambda’ and comapring predictions with target, summing the results together (below):</p>
<pre><code class="lang-auto">        correct += (lam * predicted.eq(targets_a.data).cpu().sum().float()
                    + (1 - lam) * predicted.eq(targets_b.data).cpu().sum().float())
</code></pre>
<p>But <code>v2.MixUp(num_classes=NUM_CLASSES)</code> does not return these values.</p>
<p>Thanks a lot.</p>
<pre><code class="lang-auto">from torchvision.transforms import v2
from torch.utils.data import DataLoader

NUM_CLASSES = 10

dataloader = DataLoader(dataset, batch_size=4, shuffle=True)


mixup = v2.MixUp(num_classes=NUM_CLASSES)

for images, labels in dataloader::
            inputs = images.to(device)
            targets = labels .to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward
            # track history if only in train
            with torch.set_grad_enabled(True):
                outputs = model(inputs)
                loss = criterion(outputs, targets)   # where criterion = nn.CrossEntropyLoss()

                # backward + optimize
                loss.backward()
                optimizer.step()

            # statistics
            train_loss += loss.item()

            # acc =??



</code></pre>
Is this datasets audio classification notebook bugged?

Laulito: <p>In this tutorial (<a href="https://huggingface.co/docs/transformers/tasks/audio_classification" class="inline-onebox" rel="noopener nofollow ugc">Audio classification</a>) I’ve read about a pytorch audio classifiaction notebook -</p>
<p>“For a more in-depth example of how to finetune a model for audio classification, take a look at the corresponding <a href="https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb" rel="noopener nofollow ugc">PyTorch notebook</a>.”</p>
<p>but when I tried to run it in google colab something got real messy somehow and a lot of errors occured … maybe it is a library conflict.</p>
<p>The first two cells run normally, but the third cell can’t be compiled when using datasets 1.14, because the notebook login wants username + password but results in an URL error after logging in somehow …</p>
<blockquote>
<p>404 Client Error: Not Found for url: hf/api/login<br>
Error Cannot POST /api/login</p>
</blockquote>
<p>I think this is caused by notebook_login() in datsets 1.14. If I change datasets 1.14 to just</p>
<pre><code class="lang-auto">!pip install datasets
</code></pre>
<p>Then the notebook login leads to a login with just the huggingface token and after logging in no error occurs … I proceeded with the following cells but then when I try to run the sixth cell with</p>
<pre><code class="lang-auto">from datasets import load_dataset, load_metric
</code></pre>
<p>I get an error that states</p>
<blockquote>
<p>AttributeError: readonly attribute</p>
</blockquote>
<p>So I guess this error is caused by using the wrong datasets version … I can’t do the notebook login when using datasets 1.14 but it seems that at the same time I get an error when using a datasets version that is newer than 1.14 when trying to import the functions that are needed for running the notebook … can anoyone help me fix this issue?</p>
Torch.compile too slow with SDXL model

shersoni: <p>Hello,</p>
<p>The following code runs too slow if I unable torch.compile.</p>
<p>from diffusers import DiffusionPipeline, LCMScheduler<br>
from diffusers import EulerDiscreteScheduler</p>
<p>import torch<br>
import numpy as np<br>
import cv2</p>
<p>def pil_opencv_image( image ):<br>
# Convert PIL image to a numpy array<br>
npimg = np.array(image)</p>
<pre><code># Convert RGB to BGR for OpenCV
cvimg = npimg[:, :, ::-1]
return cvimg
</code></pre>
<p>model_id = “stabilityai/stable-diffusion-xl-base-1.0”<br>
lcm_lora_id = “latent-consistency/lcm-lora-sdxl”</p>
<p><span class="hashtag-raw">#pipe</span> = DiffusionPipeline.from_pretrained(model_id, variant=“fp16”)<br>
pipe = DiffusionPipeline.from_pretrained(model_id, torch_dtype=torch.float16)</p>
<p>pipe.load_lora_weights(lcm_lora_id)<br>
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)</p>
<p>pipe.unet = torch.compile(pipe.unet, mode=“reduce-overhead”, fullgraph=True)<br>
pipe.to(device=“cuda”, dtype=torch.float16)</p>
<p>prompt = “close-up photography of old man standing in the rain at night, in a street lit by lamps, leica 35mm summilux”</p>
<p>t0 = cv2.getTickCount()<br>
image = pipe(<br>
prompt=prompt,<br>
num_inference_steps=4,<br>
guidance_scale=1,<br>
).images[0]<br>
t1 = cv2.getTickCount()<br>
tp = (t1-t0)/cv2.getTickFrequency()<br>
print("Time: ", tp )</p>
<p>cvimg = pil_opencv_image(image)<br>
cv2.imshow( “Image”, cvimg)<br>
cv2.waitKey(0)</p>

marksaroufim: <p>Which torch version are you using? This has gotten better in nightlies, granted it’ll still be slow</p>
With torch.compile(mode="max-autotune")

htforfun: <p>I have tested torch.compile(“max-autotune”) and I am amazed!  It’s awesome.</p>
<p>I have found though it doesn’t work with Flash Attention.  I have to use functions to get it to work on certain parts of code.  It would be awesome if we could do this:</p>
<pre><code class="lang-auto">with torch.compile(mode="...")
    code that compiles...

code that doesn't compile

with torch.compile(mode="different mode")
    code that needs different mode
</code></pre>
<p>then we can hyper optimize without new functions for each group!</p>
<p>Any ideas?  I don’t know how hard that is because I’m not a python expert, mostly just dinking with it, but it’s pretty cool!</p>
<p>Overall great work on it.  Anybody got any ideas?</p>

marksaroufim: <p>It should work seamlessly with <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" class="inline-onebox">torch.nn.functional.scaled_dot_product_attention — PyTorch 2.1 documentation</a> just make sure to enable the flash attention flag</p>
PyTorch supported device

prateekazam: <p>Hi,</p>
<p>I have intel GPU, Can PyTorch support this? What other devices Pytorch support? Calling torch.device with the wrong value is giving below error.</p>
<p>Expected one of cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu, xla device type.</p>
<p>Can I use the device from above?</p>

vaisakh_m: <p>No, PyTorch only supports CUDA enabled devices(Nvidia GPUs) as GPUs.<br>
You can still run PyTorch on your CPU.</p>
<aside class="quote no-group" data-username="prateekazam" data-post="1" data-topic="103987">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ed8c4c/48.png" class="avatar"> prateekazam:</div>
<blockquote>
<p>Expected one of cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu, xla device type</p>
</blockquote>
</aside>
<p>All the devices mentioned here are compatible with PyTorch.</p>
<p>Also, check if <a href="https://software.intel.com/content/www/us/en/develop/articles/getting-started-with-intel-optimization-of-pytorch.html#:~:text=Users%20can%20easily%20get%20PyTorch%20from%20its%20official%20website.&amp;text=To%20run%20PyTorch%20on%20Intel,acceleration%20support%20enabled%20by%20default." rel="noopener nofollow ugc">this</a> is useful.</p>

Mahdi_Amrollahi: <p>What about Apple Silicon or AMD or Intel Arc? Can the pytorch leverage these GPUs?</p>

marksaroufim: <p>I don’t believe the binaries are upstreamed in pytorch yet but you can find your way poking around here <a href="https://www.intel.com/content/www/us/en/developer/articles/technical/stable-diffusion-with-intel-arc-gpus.html#gs.1nozjh" class="inline-onebox">Stable Diffusion with Intel® Arc™ GPUs Using PyTorch and Docker</a></p>