lisa-lab/pylearn2 — pull request #685 by carriepl (2014-03-11 12:56:56+00:00)
Title: Local cache of datasets

Fixes gh_489
This pull request reuses much of Li's code from his own pull request. See gh-96.

Main differences with the code in his pull request are :
-Locks are managed at the file level instead of dataset level
-Processes now use readlocks to show they are using certain files. This will allow to eventually have an automated process to scan the ${PYLEARN2_LOCAL_DATA_PATH} folder and clean up unused files to free space if needed.
-The object now only caches the data and doesn't try to load it assuming it's a pickled file. This allows to easily accomodate other types of files. It also allows the caller to cache files in a different order than it tries to load them (for example if it wants some files to have a higher priority to be cached than others in the case where very little space is available on the hard disk)
-The load_file method returns an updated filename to be used by the caller to access the data instead of the data itself. It returns a local path if the file was cached and the original path otherwise.