write past end to improve speed

Summary: This makes builds 3% faster

Test Plan: tested and benchmarked

Reviewed By: andrewjcg@fb.com

FB internal diff: D1250742
Fix for bug in line numbers

Summary:
warp fails to count lines properly for this file:

struct A {
/* aaa
* bbb */
void f() {
}
};

Code after preprocessing:

struct A {

void f() {
}
};

After this diff:

struct A {

void f() {
}
};

Test Plan: as above

Reviewed By: andrewjcg@fb.com

FB internal diff: D1251072
indents are 4 spaces

Test plan: built warp

Reviewed By: andrei.alexandrescu@fb.com
.init() => .initialize()

Test plan: built warp

Reviewed By: andrei.alexandrescu@fb.com
switch to table lookup for identifier chars

Test plan: got 2% end-to-end speed improvement

Reviewed By: andrei.alexandrescu@fb.com
uint -> size_t

Summary:
title

Test Plan: unittest

Reviewed By: plchen@fb.com

FB internal diff: D1252570
refactor isContext

This also includes deployment of pull requests 1-7.

Reviewed By: andrei.alexandrescu@fb.com

Test plan: built
warp: emit gnu-style heading for preprocessed file

Summary:
gcc emits a few lines at the beginning of each preprocessed file, in the format:

`# 1 "path/to/file.cpp"`
`# 1 "/home/johndoe/code//"`
`# 1 "<command-line>"`
`# 1 "path/to/file.cpp"`

Note the two slashes and the verbatim "<command-line>". These are not always emitted, e.g. running cpp or gcc -E would emit a slightly different format. This whole thing is undocumented to the best of my ability to use google.

Anyhow, per discussion in the hhvm project, it seems that gdb has trouble locating sources with warp-preprocessed files. The only place the absolute path is emitted by gcc is the second line in the format above, so it stands to reason that that's what informs gdb. This diff makes warp output four lines in the same format as above.

Test Plan: built a couple projects, at least there were no adverse effects

Reviewed By: oyamauchi@fb.com

FB internal diff: D1257087
warp: add support for ancient #ident directive

Summary: Had no idea this even existed, but apparently it does and is used in some third-party libraries...

Test Plan: tried on small file

Reviewed By: njormrod@fb.com

FB internal diff: D1261401
warp: don't cache in memory files that use #pragma once or #include guards (by Walter Bright)

Summary: By default warp keeps in memory all included files on the off chance they will be included again. However, this is unnecessary for files that use #pragma once or #include guards - their contents is highly unlikely to be ever needed again after the first use (unless user code #undef-ines the #include guard macro). This diff reduces memory consumed by dmd by freeing the memory used by such files.

Test Plan: tried with a few projects

Reviewed By: andrewjcg@fb.com

FB internal diff: D1265618
Revert "warp: don't cache in memory files that use #pragma once or #include guards (by Walter Bright)"

Summary: This reverts commit that caches files

Test Plan: someone else did it

Reviewed By: njormrod@fb.com

FB internal diff: D1267167
warp: minor error in merging code from Walter

Summary: This has been a transcription error when I merged Walter's code by hand.

Test Plan: built a few internal projects

Reviewed By: robbert@fb.com

FB internal diff: D1270693
warp: make ptext an array instead of a bald pointer

Summary: Change authored by Walter Bright

Test Plan: built a few projects

Reviewed By: njormrod@fb.com

FB internal diff: D1270777
Small string optimization by Walter Bright

Summary: added sso

Test Plan: built stuff with it

Reviewed By: njormrod@fb.com

FB internal diff: D1270898
warp: fix in macroScanArgument

Summary: by Walter Bright

Test Plan: test suite

Reviewed By: andrewjcg@fb.com

FB internal diff: D1271859
warp: refactoring by Walter Bright

Summary: refactor predefined -> pushPredefined

Test Plan: built stuff

Reviewed By: njormrod@fb.com

FB internal diff: D1271054
warp: Leave CR characters in

Summary: This is needed mostly for Windows.

Test Plan: built stuff

Reviewed By: njormrod@fb.com

FB internal diff: D1271107
warp: change sentinel to LF

Summary: merging changes by Walter Bright

Test Plan: built stuff, will also test with all projects

Reviewed By: njormrod@fb.com

FB internal diff: D1271400
warp: avoid copy for non \ lines

Summary: by Walter Bright

Test Plan: built projects

Reviewed By: njormrod@fb.com

FB internal diff: D1271550
warp: fix line generation; improve memory allocation

Summary: by Walter Bright

Test Plan: build suite

Reviewed By: andrewjcg@fb.com

FB internal diff: D1271945