lisa-lab/pylearn2 — pull request #1236 by GeertLitjens (2014-11-13 22:13:14+00:00)
Title: Added cache_size option to HDF5 dataset

This allows you to set the chunk cache size for the HDF5 file. This is useful when you are working with very large datasets and training performance becomes limited due to diskIO. When you store your HDF5 with large chunks (e.g. 10000 samples) and using a sequential iterator, the chunk will be loaded in memory if it fits the cache. When it is in memory, copying the batches to the GPU can happen through memory and not via disk (you only need to go back to disk for the next set of 10000 samples). This is in reference to the discussion in:

https://groups.google.com/forum/#!topic/pylearn-users/J1sfGYrTMiw