Split dataset in PyTorch for CIFAR10, or whatever

Ohm: <p>How to split the dataset into 10 equal sample sizes in Pytorch?<br>
The goal is to train on each set of samples individually and aggregate their gradient to update the model for the next iteration.</p>

mrshenli: <p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for data loader</p>

pritamdamania87: <p>You can use the <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/distributed.html#DistributedSampler" rel="noopener nofollow ugc">DistributedSampler</a> for this.</p>

Ohm: <p>How we can split 60,000 data(MNIST) into 10 parts in which the first part(data1) contains 6000 data, and the second part(data2) contains 6000 and so on. So how DistributedSampler will be used for this?</p>

Ohm: <p>Below is the code in python using TensorFlow.</p>
<p>from keras.datasets import mnist<br>
from keras.models import Sequential<br>
from keras.layers import Dense<br>
from keras.layers import Dropout<br>
from keras.utils import np_utils</p>
<p>(X_train, y_train), (X_test, y_test) = mnist.load_data()</p>
<p>%worker_X[i] contains the i’th part of training data’s features.<br>
%worker_y[i] contains the i’th part of training data’s label<br>
worker_X = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
worker_y = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<aside class="quote no-group" data-username="Ohm" data-post="4" data-topic="100708" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ohm/48/29674_2.png" class="avatar"> Ohm:</div>
<blockquote>
<p>How we can split 60,000 data(MNIST) into 10 parts in which the first part(data1) contains 6000 data, and the second part(data2) contains 6000 and so on. So how DistributedSampler will be used for this?</p>
</blockquote>
</aside>
<p>% dataset size is 60000 . We want to split among 10 workers.<br>
Batch = 60000//10</p>
<p>for i in range(10):<br>
worker_X.append(X_train[i*Batch: Batch+i*Batch])<br>
worker_y.append(y_train[i*Batch: Batch+i*Batch])</p>
<p>I am looking for the same thing in PyTorch.</p>

pritamdamania87: <p><a class="mention" href="/u/ohm">@ohm</a> The AWS tutorial goes over how to use the DistributedSampler to split your dataset into even parts: <a href="https://pytorch.org/tutorials/beginner/aws_distributed_training_tutorial.html#initialize-dataloaders" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/aws_distributed_training_tutorial.html#initialize-dataloaders</a>. This is not an official PyTorch tutorial, but the “With multiprocessing” section in <a href="https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html" rel="noopener nofollow ugc">https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html</a> describes how to use DistributedSampler to do this for MNIST.</p>

Ohm: <p>Thanks.<br>
‘The AWS tutorial goes over how to use the DistributedSampler to split your dataset into even parts: <a href="https://pytorch.org/tutorials/beginner/aws_distributed_training_tutorial.html#initialize-dataloaders" rel="noopener nofollow ugc">pytorch.org/tutorials/beginner/aws_distributed_training_tutorial.html#initialize-dataloaders</a>’<br>
In this example, I did not see such a thing. The ‘num_workers=workers’ in this example is different from what I am looking for. As I mentioned I am looking for split data among workers. the fixed number of data in each worker! And I need to access them when I call a specific worker.<br>
‘<a>https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html</a>’ I saw it before, again it is not clear how it split data among workers!?! Even it is not clear that it did it or</p>

Ohm: <p>A simple example to split dataset among n_workers:</p>
<p>import torch<br>
import torch.nn as nn<br>
import torchvision<br>
import torchvision.transforms as transforms<br>
import numpy as np<br>
import time<br>
from worker import worker<br>
from worker_new import worker_new</p>
<p><span class="hashtag">#the</span> number of workers here:<br>
n_workers = 4</p>
<p>testset = torchvision.datasets.CIFAR10(root=’./data’, train=False, download=True, transform=transforms.ToTensor())</p>
<p><span class="hashtag">#Split</span> testset, you can access the data from worker 1 with Worker_data[1], and so on.<br>
temp = tuple([len(testset)//n_workers for i in range(n_workers)])<br>
Worker_data = torch.utils.data.random_split(testset, temp)</p>

pritamdamania87: <p><a class="mention" href="/u/ohm">@Ohm</a> My apologies for the previous reply not being clear. The splitting automatically occurs within the DistributedSampler based on the rank that you provide. <a href="https://github.com/pytorch/pytorch/blob/a69910868a5962e2d699c6069154836e262a29e2/torch/utils/data/distributed.py#L92-L117" rel="noopener nofollow ugc">This piece of code</a> in the DistributedSampler would illustrate how this splitting is done per worker based on the rank.</p>