How do I create custom dataset for KDD99 that can be input for Dataloader()

username: MikeT
cooked: <p>Hi, guys<br>
I am doing a project related to KDD99 dataset. After some data processing, I was able to represent the dataset as a matrix with (125973,123) dimension with ‘label’ of 5 classifications(range 0-4). After normalization, I get a matrix shown below:(each row is treated as one training example, each column is treated as one feature(excluding ‘label’ column))<br>
Then I found some existing code from other project that I want to use, but the project is dealing with FashionMNIST dataset which is a built-in dataset that can be directly accessed using interface functions.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a6ebb9f233bc9c06bfec3ae9f632c481f2b7b96.png" data-download-href="https://discuss.pytorch.org/uploads/default/0a6ebb9f233bc9c06bfec3ae9f632c481f2b7b96" title="Capture1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0a6ebb9f233bc9c06bfec3ae9f632c481f2b7b96.png" alt="Capture1" data-base62-sha1="1ui1aqBVP7zytGPNoYFfPau4mXQ" width="625" height="500" data-dominant-color="494949"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture1</span><span class="informations">984×786 20.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
So I am kinda stuck and don’t know how to code to load my dataset to Dataloader() in same fashion so that I can use existing code.<br>
I’d really appreciate if someone can point me the right direction.</p>

username: beaupreda
cooked: <p>Hello, what you need to do is implement your own dataset using the <code>torch.utils.data.Dataset</code> like this small example:</p>
<pre><code class="lang-auto">class CustomDataset(torch.utils.data.Dataset):
    def __init__(self, inputs, labels):
        self.inputs = inputs
        self.labels = labels

    def __getitem__(self, index):
        return self.inputs[index, :], self.labels[index]

    def __len__(self):
        return len(self.input)

inputs = torch.rand(125973, 122, dtype=torch.float32)
labels = torch.rand(125973, dtype=torch.float32)
dataset = CustomDataset(inputs, labels)
train_loader = torch.utils.data.DataLoader(dataset, batch_size=32, shuffle=True)
for binputs, blabels in train_loader:
    print(binputs)
    print(blabels)
</code></pre>
<p>Basically, you inherit from the PyTorch <code>Dataset</code> class and you implement the <code>__getitem__</code> and <code>__len__</code> methods. <code>__getitem__</code> is called to extract batch tensors and <code>__len__</code> determines the number of elements in the dataset. After that, use the PyTorch <code>DataLoader</code> like in your example with your own dataset and everything should work fine!</p>
<p>The following site also shows a good example of how to use the <code>Dataset</code> class of PyTorch.<br>
<a href="https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel" class="onebox" target="_blank" rel="nofollow noopener">https://stanford.edu/~shervine/blog/pytorch-how-to-generate-data-parallel</a></p>
<p>Hope this helps!</p>
<p>(Sorry for the edit, I accidently replied half an answer)</p>