uci-cbcl/genomix — pull request #47 by jakebiesinger (2013-10-31 06:46:49+00:00)
Title: Simplify and robustify cluster startup and config

@anbangx @JavierJia @Nan-Zhang

These changes allow us to run a hyracks cluster at the same time as a pregelix cluster and greatly simplify the user's interaction with the `*.properties` files.

Config file changes
----------------------------
Previously, a single config file was used to specify ports, working dirs, and logs for the currently active cluster. Rather than force the user to maintain two different config files (one for hyracks and one for pregelix), we now use a single, simplified config file but *automatically generate* the two cluster-type specific configuration files required.

The simplified file is still under `conf/cluster.properties` though we may want to change the name now that its values are incompatible with other cluster.properties we've seen so far. This file contains:

* `$WORKPATH`: the root for temporary jars and logs; also the working directory for the cc's and nc's. The four working directories for the CC's and NC's are now `$WORKPATH/hyracks/cc` ... `$WORKPATH/pregelix/nc`.
* `$DISKS`: a comma-separated list of folders corresponding to different physical disks. This is *like* `IO_DIRS` and `store` but is not the same-- the number of disks has no relationship to the number of partitions/workers/threads in use. Rather, the user-prescribed `-threadsPerMachine` will round-robin over `$DISKS` such that multiple partitions may be active on the same disk (with low `-threadsPerMachine` and high `$DISKS`, some disks may not have any partitions on them at all). `IO_DIRS` (and `store` for pregelix) are now placed under each of the `$DISKS` as `$DISKS[0]/hyracks/io_dir-0` through `$DISKS[maxThreads % len($DISKS)]/hyracks/io_dir-$maxThreads`.
* Standard ports for the CC (client, cluster and http), plus CC_DEBUG and NC_DEBUG (which are used to allow Eclipse connections to the CC and NC). Notice there are separate entries now for HYRACKS and PREGELIX.
* `NC_MAX_MEMORY`: self-explanatory, I think...

The corresponding config files are not generated until the `startCluster.sh` script is called. Normally, this is done when `genomix` is run, but you can also start a cluster independently and specify `-useExistingCluster`.

Driver changes
----------------------
* `-threadsPerMachine ` (defaults to 1)
* `-useExistingCluster` will skip startup and shutdown of the CC's and NC's. The existing cluster must still use the ports specified in `conf/cluster.properties`.
* removed `-followsGraphBuild` (not necessary anymore, thanks to Anbang's refactoring the pregelix input formats)
* removed `-clusterWaitTime` as the wait is not handled inside of the `startCluster.sh` scripts.
* both pregelix and hyracks CC's and NC's are started at the very beginning rather than being started/stopped during the run.
* we now detect startup errors in the `startcc.sh` and `startnc.sh` scripts-- If there is an error (e.g., `Address already in use`), it will be printed to the command line and driver execution will stop immediately.

Logging changes
-------------------------
* `logging.properties` has been broken into two separate files, `driver.logging.properties` and `worker.logging.properties. The former is used by the driver (where you don't want to see INFO message from hadoop) whereas the latter is for the CC and NC (where finer-grained messages are more helpful).
* I've also fixed the problems we had where you had to specify the absolute path of the logging file for the NC and CC to pick them up.
* We also no longer need the ugly static initialization block `loadLoggingFile`-- changes to the `pom.xml` fixed that.
* I still don't have a solution for the driver `%h/genomix-%u.log` file we use often when debugging-- it will still be written to your home directory.

Maven changes
-----------------------
* I've stripped away `src/main/resources/conf` and `src/main/resources/scripts` everywhere except for `genomix-driver` to reduce confusion surrounding very similarly-named scripts that are now out of date.
* A lot more of the configuration occurs in the driver's `pom.xml`, including setting the logging files and allowing the CC and NC's each have their own `app.home` directory while still sharing a single `lib`.
* I fixed a bug where the `binary-assembly` folder would sometimes be missing scripts.
* using `appassembler` as your base directory is now deprecated in favor of using `genomix-driver-2.10-SNAPSHOT` (notice the lack of `binary-assembly`-- I removed that suffix).