jdurbin/wekaMine — Issue #25: Speed up WekaMineConfigSlurper
Author: jdurbin
Created: 2012-02-21 22:36:58+00:00
State: open
Labels: New Feature, Priority Medium

WekaMineConfigSlurper is a bit slow for some uses. Specifically, when passing the config to a cluster job, I tend to assume that reading the config will take < 1 minute and schedule only a few experiments per job. However, a recent config file took 6 minutes (I believe it was about 350,000 experiments), which starts to be long enough that I'd need to up the number of experiments per job to pay for this start-up overhead. However, more experiments per job makes the process more unstable since one bad experiment can hold up a larger chunk.

So, all in all, it'd be nice if WekaMineConfigSlurper were faster. No attention to performance was paid at all, and it solves the substitution problem by brute force currently, so there should be both algorithmic improvements as well as translation into Java that is possible.