orlyatomics/orly — pull request #101 by cmaloney (2014-06-11 21:22:01+00:00)
Title: Jhm

This patchset implements JHM,

Two other PRs should land before this one (Or this one will implicitly land them), and one misc. task
- [ ] #99 - Changes to our generic libraries.
- [ ] #100 - Changes to ``
- [ ] Update Jenkins (Has to happen at the same time as the merge)

#99 and #100 are pulled out because they should be reviewed more closely for not breaking old code than the full implementation of JHM needs to be / are relatively atomic.

## Core components

- `jhm/job`: Describes some command that needs to be run, and keeps track fo both of what it needs as input, anything special it needs to run, and getting the command to run it.
- `jhm/env`: Maintains ownership / the collection of all jobs and files. Also maintains the job and file factories.
- `jhm/work_finder`:
- `jhm/job_runner`: A subprocess running queue. Rather than have n worker threads.

## Major design decisions

- There's only one thread which modifies jobs / files / etc. This eliminates a lot of locking.
- We just manage a pool of subprocesses rather than have n worker threads. See `jhm/job_runner`
- Configuration is all JSON based. See `jhm/config`.
- Files are dumb. A file is simply an entity in the JHM tree / container that lists various properties of the file (path, is source, etc). They don't know what produces them or what goes to output. Their main job is to exist, and to allow easy access to the config / various properties of the file.
- Jobs don't know about eachother. They can add attributes to their output files (Ex: list of includes as computed configuration of a file) to pass around configuration but that is it. There is no walking around the job graph.
- Work finder isolates keeping track of what we need to run to produce the requested files. It maintains sets of jobs which are currently Finished, Running (Submitted to the job runner), and Waiting (Needs 1+ other jobs to finish first). As jobs finish, it process the results of the job and asks a job if it's complete. If it is, it adds it to the Finished set, then queues any jobs which were waiting on it.

What all of this gives us is one foreground thread which queues jobs for what they need / does some fairly simple set computation to queue jobs inside a JobRunner, and then process the results (See `TWorkFinder::FinishAll`). A background thread which keeps as many subprocesses running as we have our worker count to, reading off a queue of jobs provided by the foreground thread, and storing back results for the foreground thread to grab in batches. There is a helper thread for the subprocess IO pumping (As is needed by `base/subprocess.h` in general).

## Things left TODO
There are lots of various todos in the code. Some higher-level ones:

- Implement mixin / stacked config loading
- Make an overarching generic project config which debug/release both inherit from by default (So they can share target listings, common compiler flags, etc)

## Known issues:
- If you have inlcuded by , you compile bar and the compilation completes successfully / you don't get any link errors, and then you add a foo.cc, the foo.cc won't be found / bar won't be recompiled. In this case just touch foo.h in order to get the compile to work. I don't think this is a major limitation, as if you needed foo.cc in the first place, you would have gotten link errors compiling bar. Note if you explicitly ask for foo.o, foo.o will be compiled. Compiling bar still won't force a link against it, but if bar doesn't get link errors without it, then whatever you defined in the cc wasn't needed for the compilation. Example follows if you want to see the behavior:
```
$touch foo.h
$touch bar.cc
# Have bar.cc include . Build bar.
$jhm --print-cmd bar
make_dep_file /home/firebird347/projects/src/bar.cc /home/firebird347/projects/out/debug/bar.cc.dep -std=c++1y -Wall -Werror -Wextra -Wold-style-cast -Wno-type-limits -Wno-parentheses -Wno-delete-non-virtual-dtor -msse2 -pthread -DTEST_OUTPUT_DIR="/tmp/" -DORLY_HOST -D_WEBSOCKETPP_CPP11_STL_ -isystem third_party/websocketpp/ -DDEBUG -g -I/home/firebird347/projects/src -I/home/firebird347/projects/out/debug -D'SRC_ROOT="/home/firebird347/projects/src/"'
make_dep_file /home/firebird347/projects/src/bar.cc /home/firebird347/projects/out/debug/bar.cc.dep -std=c++1y -Wall -Werror -Wextra -Wold-style-cast -Wno-type-limits -Wno-parentheses -Wno-delete-non-virtual-dtor -msse2 -pthread -DTEST_OUTPUT_DIR="/tmp/" -DORLY_HOST -D_WEBSOCKETPP_CPP11_STL_ -isystem third_party/websocketpp/ -DDEBUG -g -I/home/firebird347/projects/src -I/home/firebird347/projects/out/debug -D'SRC_ROOT="/home/firebird347/projects/src/"'
g++ -o/home/firebird347/projects/out/debug/bar.o bar.cc -c -std=c++1y -Wall -Werror -Wextra -Wold-style-cast -Wno-type-limits -Wno-parentheses -Wno-delete-non-virtual-dtor -msse2 -pthread -DTEST_OUTPUT_DIR="/tmp/" -DORLY_HOST -D_WEBSOCKETPP_CPP11_STL_ -isystem third_party/websocketpp/ -DDEBUG -g -I/home/firebird347/projects/src -I/home/firebird347/projects/out/debug -D'SRC_ROOT="/home/firebird347/projects/src/"'
g++ -o /home/firebird347/projects/out/debug/bar -rdynamic -Wl,--hash-style=gnu -Wl,--no-copy-dt-needed-entries -Wl,-z,relro -Wl,--no-as-needed /home/firebird347/projects/out/debug/bar.o -luuid -pthread -lrt -ldl -laio -lgmp -lgmpxx -lz -lreadline -lsnappy -lboost_system
[3/3] waiting: 0
# Doesn't do anything / rebuild bar.
$touch foo.cc
$jhm --print-cmd bar
[3/3] waiting: 0
$jhm foo.o
[2/2] waiting: 0
# Explicitly rebuilding foo.o. Still doesn't make bar link against it / rebuild.
$jhm --print-cmd bar
[3/3] waiting: 0
# Touching foo.h the makes everything happy
$touch foo.h
$jhm --print-cmd bar
make_dep_file /home/firebird347/projects/src/bar.cc /home/firebird347/projects/out/debug/bar.cc.dep -std=c++1y -Wall -Werror -Wextra -Wold-style-cast -Wno-type-limits -Wno-parentheses -Wno-delete-non-virtual-dtor -msse2 -pthread -DTEST_OUTPUT_DIR="/tmp/" -DORLY_HOST -D_WEBSOCKETPP_CPP11_STL_ -isystem third_party/websocketpp/ -DDEBUG -g -I/home/firebird347/projects/src -I/home/firebird347/projects/out/debug -D'SRC_ROOT="/home/firebird347/projects/src/"'
make_dep_file /home/firebird347/projects/src/bar.cc /home/firebird347/projects/out/debug/bar.cc.dep -std=c++1y -Wall -Werror -Wextra -Wold-style-cast -Wno-type-limits -Wno-parentheses -Wno-delete-non-virtual-dtor -msse2 -pthread -DTEST_OUTPUT_DIR="/tmp/" -DORLY_HOST -D_WEBSOCKETPP_CPP11_STL_ -isystem third_party/websocketpp/ -DDEBUG -g -I/home/firebird347/projects/src -I/home/firebird347/projects/out/debug -D'SRC_ROOT="/home/firebird347/projects/src/"'
g++ -o/home/firebird347/projects/out/debug/bar.o bar.cc -c -std=c++1y -Wall -Werror -Wextra -Wold-style-cast -Wno-type-limits -Wno-parentheses -Wno-delete-non-virtual-dtor -msse2 -pthread -DTEST_OUTPUT_DIR="/tmp/" -DORLY_HOST -D_WEBSOCKETPP_CPP11_STL_ -isystem third_party/websocketpp/ -DDEBUG -g -I/home/firebird347/projects/src -I/home/firebird347/projects/out/debug -D'SRC_ROOT="/home/firebird347/projects/src/"'
g++ -o /home/firebird347/projects/out/debug/bar -rdynamic -Wl,--hash-style=gnu -Wl,--no-copy-dt-needed-entries -Wl,-z,relro -Wl,--no-as-needed /home/firebird347/projects/out/debug/foo.o /home/firebird347/projects/out/debug/bar.o -luuid -pthread -lrt -ldl -laio -lgmp -lgmpxx -lz -lreadline -lsnappy -lboost_system
[5/5] waiting: 0
```