Remarkup - add underline rule

Summary: ...and a test even!

Test Plan: tests passed

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7995
underline rule - tweak it a bit

Summary:
basically what you told me to do in D7995, but I also added a "/" to the 'don't capture' clause. In my chosen test code without this check we would end up underlining "and also", and I figured this special case will probably need to handle multiple files in a row with this formatting.

Of course, this means __/something__ also won't be underlined.

Test Plan: plugged the regex into a handy online tool (http://regex101.com/) and played with it a bunch. re-ran unit tests and they passed

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7999
Make directory fixture script slightly more modern and usable

Summary:
Ref T4327. I made some small changes to this script while developing parser unit tests. Mostly:

  - Add `pht()`.
  - Use `PhutilConsole`.
  - Throw `PhutilArugmentUsageException` for questionable usage.
  - Add a `--read-only` flag to ignore changes on exit.
  - Show the CWD after unpacking the fixture, which makes it easier to `svn checkout file://...` it.
  - Improve some language.

Test Plan: Tested all the features; used these features while working on T4327 to good effect.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D8021
Add unit test scaffolding for XHPAST/PHPAST

Summary:
Ref T4334. Allows us to build tests against this so we can accomplish goals described in T4334 with greater confidence.

This allows three types of tests to be defined:

  - `pass`: Input is expected to parse into a specific tree.
  - `fail-syntax`: Input is expected to fail with a syntax error.
  - `fail-parse`: Input is expected to parse the wrong tree.

The latter two test types allow us to write passing tests of broken/bad behavior while keeping it clear which tests are passing-and-good and which are passing-but-broken.

Test Plan: Ran unit tests.

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4334

Differential Revision: https://secure.phabricator.com/D8025
Add PHPAST test cases for most lexer patterns

Summary: Ref T4334. Adds test cases for most of the lexer rules: tags, literals, and comments. Remaining lexer rules are mostly difficult to test in isolation, or have a known issue in the case of HEREDOC.

Test Plan: The JSON dumps aren't very human readable, so I'm verifying that the parser output is sane using the web UI PHPAST tool before committing it to tests.

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4334

Differential Revision: https://secure.phabricator.com/D8026
Add failing PHPAST test case for binary literals

Summary: Ref T4334. We do not currently support these, but should. Support will turn this into a "pass" test case.

Test Plan: Generated expected output with similar file using a hex literal, ran unit tests.

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4334

Differential Revision: https://secure.phabricator.com/D8027
Add a PHPAST test case for most of the PHP grammar

Summary: Ref T4334. This test case covers most of the common PHP grammar rules. It's not exhastive and avoids some of the tricky parts, but provides a wide base of rule tests.

Test Plan: Examined parse manually in PHPAST web tool. Ran unit tests.

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4334

Differential Revision: https://secure.phabricator.com/D8028
Add PHPAST test for empty heredoc

Summary: Ref T4334. Ref T3512. This result is wrong, but cover it with a test before I muck around with the parser itself.

Test Plan: Ran unit tests.

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T3512, T4334

Differential Revision: https://secure.phabricator.com/D8029
Add PHPAST tests for static class expressions, callable, traits, finally and short array syntax

Summary: Ref T4334. These are all failing, but should pass. They pass with `php -l` on modern PHP.

Test Plan: Ran unit tests.

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4334

Differential Revision: https://secure.phabricator.com/D8030
Add PHPAST test for remaining language features

Summary: Ref T4334. Adds yield, foreach-list, literal-deref tests. All currently fail. Adds a heredoc terminal test.

Test Plan: Ran unit tests.

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4334

Differential Revision: https://secure.phabricator.com/D8031
Align PHPAST lexer to PHP 5.5.8

Summary:
Ref T4334. This aligns the lexer (but not the parser) with PHP 5.5.8. It also applies the patch on T3512. This has these positive effects:

  - Fixes binary literals.
  - Fixes empty heredoc.
  - Corrects T_BINARY_CAST into T_STRING_CAST (consistent with 5.5.8).
  - Removes T_UNICODE_CAST (not in 5.5.8).
  - Recognizes keywords `trait`, `__trait__`, `yield`, `insteadof`, `callable`.

...these sideways effects:

  - Slightly improves terminal heredocs (previously: parse error; now: bad parse).

...and these negative effects:

  - Breaks the `callable` test, since `callable` is now recognized as a keyword but the parser doesn't yet support it.

Test Plan: Rebuilt xhpast, ran unit tests.

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4334

Differential Revision: https://secure.phabricator.com/D8032
Mostly align parser to PHP 5.5.8

Summary: Ref T4334. Adds support for: traits; yield; short array syntax; callable; finally.

Test Plan:
Unit tests. Attempted to use PHPAST web view to verify I didn't build the trees wrong.

{F103626}

Generally, these changes are difficult to verify by inspection, and errors only become clear once the tree is in use. So probably not everything is quite correct, but it should be relatively close.

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4334

Differential Revision: https://secure.phabricator.com/D8033