Dataloader with Numpy much slower when num_workers > 0

XavierB: <p>Hello everyone,</p>
<p>I have been working on a project where the data and features are stored in Numpy arrays, and I found that the <strong>DataLoader was quite slow</strong> when the <code>num_workers</code> &gt; 0, so I decided to recreate the issue with a dummy example:</p>
<pre><code class="lang-auto">import numpy as np
from torch.utils.data import DataLoader


class NumpyDataset:
    def __init__(self, size: int):
        self.data = np.random.rand(size, 2)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, i):
        return {
            "feature": self.data[i][0],
            "target": self.data[i][1]
        }

SIZE = #TBD
BATCH_SIZE = #TBD
NUM_WORKERS = #TBD

ds = NumpyDataset(size=SIZE)
dl = torch.utils.data.DataLoader(dataset=ds, batch_size=BATCH_SIZE, num_workers=NUM_WORKERS)
for _ in dl:
    pass
</code></pre>
<p>After running some benchmarks for different combinations of parameters, these were the results:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:right">size</th>
<th style="text-align:right">batch_size</th>
<th style="text-align:right">num_workers</th>
<th style="text-align:right">total_time (s)</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:right">1e+05</td>
<td style="text-align:right">32</td>
<td style="text-align:right">0</td>
<td style="text-align:right">0.041524</td>
</tr>
<tr>
<td style="text-align:right">1e+05</td>
<td style="text-align:right">32</td>
<td style="text-align:right">8</td>
<td style="text-align:right">0.387215</td>
</tr>
<tr>
<td style="text-align:right">1e+05</td>
<td style="text-align:right">64</td>
<td style="text-align:right">0</td>
<td style="text-align:right">0.0182004</td>
</tr>
<tr>
<td style="text-align:right">1e+05</td>
<td style="text-align:right">64</td>
<td style="text-align:right">8</td>
<td style="text-align:right">0.260331</td>
</tr>
<tr>
<td style="text-align:right">1e+05</td>
<td style="text-align:right">128</td>
<td style="text-align:right">0</td>
<td style="text-align:right">0.0164798</td>
</tr>
<tr>
<td style="text-align:right">1e+05</td>
<td style="text-align:right">128</td>
<td style="text-align:right">8</td>
<td style="text-align:right">0.184617</td>
</tr>
<tr>
<td style="text-align:right">1e+06</td>
<td style="text-align:right">32</td>
<td style="text-align:right">0</td>
<td style="text-align:right">2.30033</td>
</tr>
<tr>
<td style="text-align:right">1e+06</td>
<td style="text-align:right">32</td>
<td style="text-align:right">8</td>
<td style="text-align:right">28.3145</td>
</tr>
<tr>
<td style="text-align:right">1e+06</td>
<td style="text-align:right">64</td>
<td style="text-align:right">0</td>
<td style="text-align:right">1.73181</td>
</tr>
<tr>
<td style="text-align:right">1e+06</td>
<td style="text-align:right">64</td>
<td style="text-align:right">8</td>
<td style="text-align:right">14.0961</td>
</tr>
<tr>
<td style="text-align:right">1e+06</td>
<td style="text-align:right">128</td>
<td style="text-align:right">0</td>
<td style="text-align:right">1.51957</td>
</tr>
<tr>
<td style="text-align:right">1e+06</td>
<td style="text-align:right">128</td>
<td style="text-align:right">8</td>
<td style="text-align:right">8.15612</td>
</tr>
<tr>
<td style="text-align:right">1e+07</td>
<td style="text-align:right">32</td>
<td style="text-align:right">0</td>
<td style="text-align:right">22.3278</td>
</tr>
<tr>
<td style="text-align:right">1e+07</td>
<td style="text-align:right">32</td>
<td style="text-align:right">8</td>
<td style="text-align:right">281.27</td>
</tr>
<tr>
<td style="text-align:right">1e+07</td>
<td style="text-align:right">64</td>
<td style="text-align:right">0</td>
<td style="text-align:right">15.7327</td>
</tr>
<tr>
<td style="text-align:right">1e+07</td>
<td style="text-align:right">64</td>
<td style="text-align:right">8</td>
<td style="text-align:right">151.014</td>
</tr>
<tr>
<td style="text-align:right">1e+07</td>
<td style="text-align:right">128</td>
<td style="text-align:right">0</td>
<td style="text-align:right">14.3264</td>
</tr>
<tr>
<td style="text-align:right">1e+07</td>
<td style="text-align:right">128</td>
<td style="text-align:right">8</td>
<td style="text-align:right">75.8562</td>
</tr>
</tbody>
</table>
</div><p>From these results I could see that:</p>
<ul>
<li>
<code>num_workers = 0</code> is around <strong>1 order of magnitude faster</strong> than <code>num_workers = 8</code>
</li>
<li>The difference appears to reduce with bigger batch sizes</li>
</ul>
<p>Does anyone know why this might be happening?<br>
Is it recommender to run single thread operations when dealing with NumPy?</p>
<p>Thanks you!</p>

ptrblck: <p>I don’t think this effect is necessarily depending on the usage of <code>numpy</code> but might be the expected overhead from using multiple processes to only index an already preloaded dataset.<br>
Multiple workers are beneficial especially if you are lazily loading and processing the data, i.e. if a single sample is loaded and transformed in each <code>__getitem__</code> call. In this case each worker will create a full batch in the background while the main thread is busy with the actual model training.<br>
In your example you have already preloaded the dataset in the <code>__init__</code> such that each worker will only index the data sample from its copy and create the batch, which could yield an overall slowdown due to the added overhead.</p>

imayachita: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
I have a similar issue despite loading and processing the data in <code>__getitem__</code> call instead of <code>__init__</code>. Is there any other possible reason why?</p>

ptrblck: <p>Assuming you are lazily loading the samples in the <code>__getitem__</code> I would guess you might see a bad perf if the actual loading is the bottleneck and thus blocks the other code parts (which could be the case if you are using e.g. a slow network drive).<br>
Could you profile the data loader alone and see how the speed is for different number of workers?</p>

imayachita: <p>I used <a href="https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html" rel="noopener nofollow ugc">torch profiler</a> and here what I got:</p>
<p><strong>Dataloaders time (microsecs) with various num workers</strong><br>
0: 1,484,033<br>
1: 13,361,716<br>
2: 14,867,360<br>
3: 14,596,160<br>
4: 15,243,254<br>
5: 14,412,214<br>
20: 15,291,521</p>

imayachita: <p>I want to understand the multiprocessing happening here further. Are the operations in <code>__init__</code> done in the main thread and only the operations inside <code>__getitem__</code> happen in each of the multiple workers? Hence, in the XavierB’s case the overhead of copying the loaded data (that happened in the <code>__init__</code>) to each worker is the bottleneck. Is it correct that <code>__init__</code> should contain as minimum operation as possible?</p>
<p>Also, after the workers load the data, will they put the loaded the data in sort of “queue” (though maybe queue is not the right term here) to be fed to GPU during training? If yes, then I suspect there will be an optimum <code>num_workers</code> because if we keep increasing that number, we will just create a long queue that is not necessary (and perhaps consumes the memory)?</p>

Saurabh_Mishra: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I am having similar problem when I am loading numpy files in Dataset. Dataloading is too slow that GPU is waiting for the data thus resulting into no GPU utilisation.</p>
<p>Here is sample dataset :</p>
<pre><code class="lang-auto">class CNNRegressionDataset(Dataset):
    def __init__(self, dataset_path,cfg):
        super(CNNRegressionDataset, self).__init__()
        self.target_dir = os.path.join(dataset_path, 'y')
        self.input_dir = os.path.join(dataset_path, 'x')

        self.cfg = cfg

    def __len__(self):
        return len(glob.glob(os.path.join(self.input_dir, "*.npy"), recursive=True))

    def __getitem__(self, idx):
        input_file = glob.glob(os.path.join(self.input_dir, "*.npy"), recursive=True)[idx]
        target_file = os.path.join(self.target_dir, os.path.basename(input_file))

        x = np.load(input_file)
        if self.cfg.DATASETS.DEPTH_ONLY:
            x = x[:,:,-1]
        if x.shape[0] != self.cfg.DATASETS.DESIRED_SIZE[0] or x.shape[1] != self.cfg.DATASETS.DESIRED_SIZE[1]:
            x = padding(x, self.cfg.DATASETS.DESIRED_SIZE, num_channels=self.cfg.DATASETS.NUM_CHANNELS)
        x = x.reshape(x.shape[0],x.shape[1],self.cfg.DATASETS.NUM_CHANNELS)
        y = torch.tensor(np.load(target_file))
        # changing mm to meter
        # if np.max(x)&gt;10:
        #     x = x/1000
        #     y = y/torch.tensor(1000000)
        # # changing meter to centimeter 
        # else:
        #     x = x*100

        if self.cfg.DATASETS.NORMALISE_DEPTH:
            x = normalize_image(x,input_file)
        x = torch.tensor(x)
        x = x.permute(2, 0, 1)
       
</code></pre>
<p>Dataset Initialisers :</p>
<pre><code class="lang-auto">train_dataset = CNNRegressionDataset(dataset_path=dataset_path,cfg=cfg)
# train_dataset = datasets.FakeData(50000, (1,224, 224), 1, transforms.ToTensor())
</code></pre>
<p>Here is dataloader :</p>
<pre><code class="lang-auto">
train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=128, shuffle=(train_sampler is None),
        num_workers=0, pin_memory=True, sampler=train_sampler)
</code></pre>
<p>Using CNNRegressionDataset, GPU util is 0 while FakeData, my GPU util is 100</p>

ptrblck: <p>For general data loading advice you can check <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a>.</p>