lisa-lab/pylearn — pull request #96 by yaoli (2012-07-06 02:52:05+00:00)
Title: assembla ticket 127, Pylearn2 dataset loader that copy to the local directory

Mainly concern with Fred and Steven about their idea of get a data_loader to pylearn2.

This ticket is half done and not thoroughly tested (have no idea how to do the test in fact). I put it here merely to let people provide some suggestions.

The main idea of this ticket:

When running mutiple jobs using the same dataset, each job would have to read
the same dataset from ${PYLEARN2_DATA_PATH}, which may cause a great burden on the network.

The solution is to copy the required dataset into ${PYLEARN2_LOCAL_DATA_ROOT} that is
local to the node that is executing the job. The job can thus read locally the dataset
if possible.

Then whenever a new job is run on a node, it searches the required dataset in
${PYLEARN2_LOCAL_DATA_ROOT}. If the dataset is there, use it directly, otherwise, copy
it from the remote server.

This way the access to NFS could be dramatically decreased by local access when it comes to datasets, especially when jobdispatch is used.