numenta/nupic — pull request #959 by mrcslws (2014-05-26 00:16:27+00:00)
Title: Make runWithConfig directory handling more resilient / functional

Fixes #919

I followed the current naming, but I think it's becoming confusing. I don't have enough perspective to pick good names, so I'll just show an example and see if anyone wants to suggest changes:

I modified hotgym to swarm like this:
```
modelParams = permutations_runner.runWithConfig(
swarmConfig,
{"maxWorkers": maxWorkers, "overwrite": True},
outputLabel=outputLabel,
outDir="outDir",
permWorkDir="permWorkDir",
verbosity=0
)
```

Afterward, here's what gets placed:
```
$ ls -l -R
./outDir:
total 18
-rwxrwxrwx 1 root root 13360 May 25 16:25 description.py
-rwxrwxrwx 1 root root 4664 May 25 16:26 description.pyc
-rwxrwxrwx 1 root root 4439 May 25 16:25 permutations.py

./permWorkDir:
total 5
drwxrwxrwx 1 root root 4096 May 25 16:26 model_0
-rwxrwxrwx 1 root root 36 May 25 16:25 rec-center-hourly_HyperSearchJobID.pkl
-rwxrwxrwx 1 root root 1100 May 25 16:26 rec-center-hourly_Report.csv

./permWorkDir/model_0:
total 0
-rwxrwxrwx 1 root root 2503 May 25 16:26 description.py
-rwxrwxrwx 1 root root 1323 May 25 16:26 description.pyc
-rwxrwxrwx 1 root root 5446 May 25 16:26 model_params.py
-rwxrwxrwx 1 root root 331 May 25 16:26 params.csv
```

Complaints I have about this naming:
- The term `outDir` is overloaded. In permutations_runner, `outDir` is often different from `options["outDir"]`.
- The "outDir" doesn't ultimately have the file that I care about (model_params.py). I still end up having to crack open the "permWorkDir" (or write the file myself from the `runWithConfig` return value, as happens in hotgym)
- The "permWorkDir" name implies that it changes the working dir. But it doesn't. This is relevant, since swarm_description.py needs to specify the file relative to the working dir (e.g. it should be `file://rec-center-hourly.csv`, but this naming implies that it should be `file://../rec-center-hourly.csv` )