Fix js highlighting
Use pygments for C++ syntax highlighting
Fix typo and line length
Don't yield anything for regions
style
Switch from a sequence of linear list lookups to a single hash lookup for a theoretical speed boost.
Use Pygments rather than our custom C++ tokenizer for syntax highlighting. Close #156.
whitespace
Catch tracebacks thrown by workers, and report them in context, not just relative to where the error was collected in concurrent.futures. Refs bug 924631.

Stop reporting incorrect file IDs of the workers that failed.
Upgrade concurrent.futures to 2.1.4.

It doesn't seem to do any harm, and there have been some bug fixes since the last version.
Fix a NameError in dxr-build.
style
Recognize includes in the compiler plugin. Rip out custom C++ tokenizer, now unused.

Don't guess them after the fact from Python.

This adds an `includes` table to the schema.

Also, tighten up some bare excepts to not catch control-Cs.
Enable git fsck.

Our new stage and prod boxes have new enough gits to support the --no-dangling option.
Wait to deploy new web app code until a new enough instance to support it has been built.
Add a diagnostic message to build.py.

Sometimes, the HTML building hangs. This time, it was before any workers completed. This will tell me whether it's hanging while submitting the jobs or whether, instead, as_completed() isn't working.
Upgrade to Pygments 1.6 so we don't have an infinite loop in the lexer while highlighting mozilla-central/gfx/harfbuzz/src/hb-ot-tag.cc.
Line-buffer the log files so we can reliably see what the last file processed by a frozen HTML worker was.
Don't explode if an ignored file gets #included.
Retire our custom C++ tokenizer in favor of slurping up includes within the clang compiler plugin. Fixes bug 924631.

Many other improvements as well:
* Better diagnostic messages for failing HTML workers
* Deployment improvements, including schema version awareness and git hash verification.
* Use Pygments for C++ syntax highlighting. (#156)
* Upgrade concurrent.futures to 2.1.4.