akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch Lightning : Number Of Training and Validation Batches

Ajinkya_Ambatwar: <p>Hi I have a custom map-style dataLoader function for my application. Please excuse the indentation below.</p>
<pre><code class="lang-auto">class data(object):
    def __init__(self, train):
            self.train = train
           &lt;some other init&gt;
   def __len__(self):
           if self.train:
                  return 640
           else:
                 return 160
  def __getitem__(self, batchIdx):
         &lt;do something&gt;
         return something
</code></pre>
<p>Now as it can be seen, in the <code>__len__</code>, I have used dummy numbers. This is for my own experimentation. I am using a batch size of 32.<br>
This is my training and validation dataloader build inside <code>LightningModule</code> class</p>
<pre><code class="lang-auto">def prepare_data(self):
        self.train_dset = data(
            train=True
        )
        self.val_dset = data(
            train=False
        )

    def _build_dataloader(self, dset, mode):
        return DataLoader(
            dset,
            batch_size =32,
            drop_last=mode == "train",
            collate_fn=collate_fn,
            pin_memory=True,
            num_workers=8,
            prefetch_factor=1
        )

    def train_dataloader(self):
        return self._build_dataloader(self.train_dset, mode="train")

    def val_dataloader(self):
        return self._build_dataloader(self.val_dset, mode="val")
</code></pre>
<p>I am expecting 20 batches for training(return len of 640 for batch size of 32) and 5 for validation(return len of 160 for batch size of 32). But during training, it prints</p>
<pre><code class="lang-auto">Epoch 0: 100%|██████████████| 25/25
Validating: 100%|██████████████| 5/5
</code></pre>
<p>The validation size is correct. But I can’t understand the reason behind 25 training batches here.<br>
Am I missing something?</p>
<p>Thank you!!</p>

tom: <p>I think this is the total number of batches (training + validation).</p>
<p>Best regards</p>
<p>Thomas</p>
Importing datasets in pytroch-geometric

Mriganka_Nath: <p><code>from torch_geometric.datasets import KarateClub</code><br>
this gives me the error</p>
<pre><code class="lang-auto">OSError: /usr/local/lib/python3.7/dist-packages/torch_sparse/_convert_cpu.so: undefined symbol: _ZN6caffe28TypeMeta21_typeMetaDataInstanceIN3c107complexIfEEEEPKNS_6detail12TypeMetaDataEv
</code></pre>
<p>I am running on colab.</p>