Pytorch performance concern

username: ecolss
cooked: <p>Since Pytorch is in Python, inevitably the code will involve some Python overhead compared C++ code.</p>
<p>For example, with the data loading utility from Pytorch, we could code up a data-loader to load our data, be it text or image etc.<br>
But what I observed is this Python data-loader is unbelievably slow than my C++ loader, since my dataset is huge, I cannot load it all into memory,so I have to iterate this dataset chunk-by-chunk, one by one epoch.</p>
<p>The slow data-loading in Python is un-tolerable, which also makes me think even if I fully load the data into memory, the network training Python code will also bring in substantial overhead.</p>
<p>BTW, in order to speed up the data-loading, should I discard this Python data-loader, and write Py-wrapper over my C++ loader?</p>

username: ptrblck
cooked: <p>Have you tried a few settings for the <code>DataLoader</code>, e.g. changing <code>num_workers</code>, using <code>pin_memory</code> etc.?</p>
<p>If so, what kind of data do you have and how large is the difference in loading time?</p>

username: ecolss
cooked: <p>Not yet, will have a try.</p>
<p>But still wonder, to match my C++ loading performance, Python <code>DataLoader</code> would have to spare more than one workers or even set these advanced options such as <code>pin_memory</code>?</p>

username: ptrblck
cooked: <p>Well, I don’t know, what kind of data you are dealing with, and how you are loading it.<br>
Based on the type of data, you could probably tune the Python code even more.</p>
<p>Using multiple processes to load your data gives you the advantage, that the next batch might be ready waiting for the GPU to finish its current operation.<br>
I’m not sure, how you are dealing with this in your C++ code, but if it’s sequential, you’ll have to wait for the data loading to finish, before you can push it onto the GPU, which might kill your speed advantage.</p>
<p>Also, with <code>pin_memory=True</code> you’ll save some unnecessary copying between pageable and pinned memory, when you would like to use the GPU.</p>

username: ecolss
cooked: <p>Well, I couldn’t help asking this, my textual data is structured, and no matter the loading is in C++ or <code>Pytorch</code>'s <code>Dataset</code>, the loaded lines have to be parsed.<br>
As tested, C++ loading (single thread) is much faster (like 10x) than the Python data-loader (single worker), maybe increase <code>num_worker</code> could help some, but do you think if it’s advisable to just plugin the C++ loader (py wrapped) into my pipeline?</p>

username: ptrblck
cooked: <p>Maybe it’ll give you some performance advantage, but you should definitely check, if it’s the bottleneck in your application.<br>
It’s always annoying to fine tune some methods when the bottleneck is somewhere else.<br>
You could have a look at <a href="https://pytorch.org/docs/master/bottleneck.html">torch.utils.bottleneck</a> to check your model code.</p>
<p>To time your <code>DataLoader</code> you could have a look at the <a href="https://github.com/pytorch/examples/blob/master/imagenet/main.py#L187">imagenet example</a>.</p>
<p>Also, how are you currently loading your text data in pytorch? It might help to save it as a hdf5 file.</p>

username: thiagocrepaldi
cooked: <p>Is it acceptable for you to implement a C++ parser that plugs to python dataloader? If so, look at  <a href="https://discuss.pytorch.org/t/how-to-use-dataset-larger-than-memory/37785/9" class="inline-onebox">How to use dataset larger than memory?</a>. The ChunkDataset API might help you and there is python bindings on the way</p>