amueller/joblib — comment on commit 4df1fabdda2b7e00a16b02848b760aa4076a96e5
File:  line 0
By amueller:

Just a short update on what I did:
Parallelizing the Forest construction over network did not work out as the memory transfer killed it. That could not be circumvented by a transformer as the data was given from the Forest to the trees.

Also for some reason the data transfer was really slow. I didn't benchmark it yet but our admin got 400MB/s using netcat. The speed I had must have been much lower. Don't know why.

Maybe a final remark: the forest tries to be smart about inter-process memory transfers and uses the "n_jobs" to create lists of jobs.
If creating each tree takes very little time, this might be important. I am not sure how well this will play with other paralellization engines.