Lay groundwork for configuration-driven linters

Summary:
Ref T2039. That task has a bunch of discussion, but basically we do a poor job of serving the midrange of lint configuration right now.

If you have something simple, the default linters work.

If you have something complex, building your own engine lets you do whatever you want.

But many users want something in between, which isn't really well accommodated. The idea is to let you write a `.arclint` file, which looks something like this:

  {
   "linters" : {
      "css" : {
        "type" : "csslint",
        "include" : "(\.css$)",
        "exclude" : "(^externals/)",
        "bin" : "/usr/local/bin/csslint"
      },
      "js" : {
        "type" : "jshint",
        "include" : "(\.js$)",
        "exclude" : "(^externals/)",
        "bin" : "support/bin/jshint",
        "interpreter" : "/usr/local/bin/node"
      }
   }
  }

...which will provide a bunch of common options around lint severity, interpreter and binary locaitons, included and excluded files, etc.

This implements some basics, and very rough support in the Filename linter.

Test Plan:
Generated a `.arclint` file and saw it apply filename lint correctly. Used `debug` mode and tried invalid regexps.

  {
    "debug" : true,
    "linters" : {
      "filename" : {
        "type" : "filename",
        "exclude" : ["@^externals/@"]
      }
    }
  }

Next steps include:

  - Provide an external linter archetype (T3186) and expose a common set of configuration here ("bin", "interpreter", "flags", "severity").
  - Provide a `.arcunit` file which works similarly (it can probably be simpler).

Reviewers: btrahan, Firehed

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2039

Differential Revision: https://secure.phabricator.com/D6797
Simplify and demuck some of the linter test cases

Summary:
Ref T3186.

  - Every linter builds a WorkingCopyIdentity in the same way, with no specialized data. Don't do that.
  - Linters get passed a goofy hardcoded ".php" path. Don't do that.
  - Linters generally run on an imaginary path, which might not work. Just give them a real path by building a tiny working copy in `/tmp`.
  - Fix a TODO now that we have better typechecking.

Test Plan: `arc unit --everything`, intentionally broke a test to make sure that still works.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T3186

Differential Revision: https://secure.phabricator.com/D6798
Rebuild map

Summary: rebuild arc map

Test Plan: run phabricator

Reviewers: epriestley, btrahan

Reviewed By: epriestley

CC: Korvin, aran

Maniphest Tasks: T3748

Differential Revision: https://secure.phabricator.com/D6804
Introduce a rough abstract base class for "linters which run programs and read the results"

Summary:
Ref T3186. We have about 50 linters which run programs and read the results, all of which have ad-hoc one-off custom config that isn't formalized anywhere.

Consolidate all this stuff into `ArcanistExternalLinter`, which is configurable through `.arclint` (although nothing supports this quite yet).

Extend CSSLint and Pep8Lint from `ArcanistExternalLinter`.

Add unit tests for both.

There are still some rough edges here, but it mostly seems to work pretty well.

Test Plan: Ran unit tests, hit some (most?) of the error cases.

Reviewers: btrahan

Reviewed By: btrahan

CC: chad, aran, Firehed

Maniphest Tasks: T3186

Differential Revision: https://secure.phabricator.com/D6800
Port flake8 to ArcanistExternalLinter

Summary:
Ref T3186. Brings another linter onboard. This one uses the stdin stuff.

The unit test was ostensibly broken so I fixed it, but that might just be some kind of version issue.

Test Plan: Unit tests.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T3186

Differential Revision: https://secure.phabricator.com/D6802
Expose PEP8, Flake8 and CSSLint engines to .arclint

Summary:
Ref T3186. Ref T2039. Allow these linters to be selected with `.arclint`.

Also allow severities to be set.

Also fix some other minor bugs.

Test Plan:
https://github.com/epriestley/arclint-examples
https://github.com/epriestley/arclint-examples/blob/master/.arclint

Reviewers: btrahan

Reviewed By: btrahan

CC: chad, aran

Maniphest Tasks: T2039, T3186

Differential Revision: https://secure.phabricator.com/D6803
Better heuristic for checking for hgsubversion

Summary:
The existing heuristic checks for the existence of .hg/svn, but it
turns out that this directory can exist in a non-svn hg repo if the user or a
script ever runs a 'hg svn' command.

Now we check for a file inside .hg/svn.  The hgsubversion maintainer said this
particular file will always be present in hgsubversion repos.

Test Plan:
arc land --trace
Verified it used 'hg push' and not 'hg push -r ...'. This indicates it
considered the repo to be an hgsubversion repo.

Reviewers: epriestley

Reviewed By: epriestley

CC: dschleimer, sid0, Korvin, aran

Differential Revision: https://secure.phabricator.com/D6807
Ready more linters and linter functions for .arclint

Summary:
Ref T3186. Ref T2039. Continues work on readying linters for `.arclint`.

  - **Ruby**: Make this an ExternalLinter.
  - **Priority**: Currently, linters have an implicit "correct" order (notably, the "NoLint" linter needs to run before other linters). Make this explicit by introducing `getLinterPriority()`.
  - **Binaries**: Currently, linters manually reject binary files. Instead, reject binary files by default (linters can override this if they do want to lint binary files).
  - **Deleted Files**: Currently, linters manually reject deleted files (usually in engines). Instead, reject deleted files by default (linters can override this).
  - **Severity**: Move this `.arclint` config option up to top level.
  - **willLintPaths()**: This method is abstract, but almost all linters provide a trivial implementation. Provide a trivial implementation in the base class.
  - **getLintSeverityMap()/getLintNameMap()**: A bunch of linters have empty implementations; these are redundant. Remove them.
  - **Spelling**: clean up some dead / test-only / unconventional code.
  - **`.arclint`**: Allow the filename, generated, nolint, text, spelling and ruby linters to be configured via `.arclint`.

Test Plan:
https://github.com/epriestley/arclint-examples/commit/458beca3d65b64d52ed612904ae66eb837118b94

Ran unit tests.

Reviewers: btrahan

Reviewed By: btrahan

CC: Firehed, aran

Maniphest Tasks: T2039, T3186

Differential Revision: https://secure.phabricator.com/D6805
Automatically answer excuse prompts if stdin is not a TTY

Summary: Fixes T3696. Currently, we abort. If stdin is not a TTY, we should just continue. A script which cares could conceivably run `arc lint` and `arc unit` separately, but it seems unlikely that any script would ever want to fail here.

Test Plan: Ran `echo -n '' | arc diff --create --verbatim` with a lint error and got a revision (D6720).

Reviewers: Firehed, btrahan, vrana

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T3696

Differential Revision: https://secure.phabricator.com/D6721
Add support for 'phpunit_binary' config setting

Summary:
If present, this will override the default phpunit path. Allows easy
integration of Composer-provided installs of phpunit (ex. set phpunit_binary to
vendor/bin/phpunit). If the path provided doesn't resolve to an executable it
will assume the path is relative to the project root.

fix line length issue

Test Plan:
Added phpunit_binary to .arcconfig in a simple project using Composer with
phpunit/phpunit package as part of require-dev (installed to
$ROOT/vendor/bin/phpunit). Phpunit not otherwise installed on the system. Set
unit.engine to PhpunitTestEngine. Confirmed that 'arc unit' used the specified
binary, both at project root and from subdirectories.

Reviewers: epriestley

CC: aurelijus, Korvin, aran

Differential Revision: https://secure.phabricator.com/D6791
When converting a file to a binary, populate the binary's data

Summary:
Currently, we prompt the user to mark non-UTF8 files as binary, but don't actually attach the data to the change when they do. This means we don't upload the data, and can't patch it later.

A simple reproduction case is to build a test file (I used one with bytes from 1..255):

  $ # Don't include \0, since Git treats that specially.
  $ ./echo_every_byte_from_1_to_255_inclusive.erl > example.txt

Then add it:

  $ git add example.txt
  $ git commit -a -m derp
  $ arc diff --only HEAD^

You'll be prompted to convert the file to binary:

  Do you want to mark this file as binary and continue? [Y/n] y

Before this patch, that would be followed by:

  Uploading 0 files...

...which is incorrect; we need to upload the new data. After this patch, this shows:

  Uploading 1 files...

...which is also incorrect, but only grammatically. Diffs created after this patch apply back cleanly with `arc patch` and restore the file properly.

Test Plan: Followed instructions above, restoring a textual binary conversion by using `arc patch`.

Reviewers: zeeg, btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D6815
Make arc patch slightly better about submodules

Summary:
Ref T3776, Ref T479. Say you have some DN, with a submodule X@Y. Later, X@Z in your working copy / repo. If you run arc patch DN, you'd end up with a dirty working copy claiming that X@Z was wrong and it should be X@Y.

To fix, basically run 'submodule init' and 'submodule update'. This makes it so after "arc patch" if you run "git status" it looks clean.

Gross part though now is if you then "git checkout master" you'll have a dirty checkout the other way. I think this is better though.

Test Plan: made a new repository where I added libphutil @ X, did some work (DX), then made libphutil @ y. When I arc patch'd DX, things looked good!

Reviewers: epriestley

Reviewed By: epriestley

CC: csilvers, Korvin, aran

Maniphest Tasks: T479, T3776

Differential Revision: https://secure.phabricator.com/D6837
Support PHPCS as a `.arclint` linter

Summary:
Ref T3186. Ref T2039. Ref T3771. A few effects here:

  # Expose PHPCS as a `.arclint` linter.
  # Turn PHPCS into an ArcanistExternalLinter linter.
  # Add test coverage for PHPCS.
  # Add a `severity.rules` option to `.arclint`. Some linters have very explicit builtin severities ("error", "warning") but their meanings are different from how arc interprets these terms. For example, PHPCS raises "wrong indentation level" as an "error". You can already use the "severity" map to adjust individual rules, but if you want to adjust an entire linter it's currently difficult. This rule map makes it easy. There's substantial precedent for this in other linters, notably all the Python linters.

For `severity.rules`, for example, this will turn all PHPCS "errors" into warnings, and all of its warnings into advice:

      "severity.rules" : {
        "(^PHPCS\\.E\\.)" : "warning",
        "(^PHPCS\\.W\\.)" : "advice"
      }

The user can use `severity` (or more rules) to get additional granularity adjustments if they desire.

Test Plan: https://github.com/epriestley/arclint-examples/commit/5bb919bc3a684d6654dcc28622de9a29ec08307d

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, ajtrichards

Maniphest Tasks: T2039, T3186, T3771

Differential Revision: https://secure.phabricator.com/D6830
Fixup lint testing for changes in D6798

Summary:
We have some linters that trigger based on the path name
in the tree (some rules apply in some dirs and not others).
The changes in D6798 caused all the paths to appear to be outside
the tree, so allow for passing a fake through from those test cases
that are sensitive to this.

We also have a test for the copyright linter, and that needs to read
settings from the .arcconfig file.  The change to faking a working
copy meant that this config option was effectively unset, so add a way
to pass the entire arcconfig through from the tests that need it.

Lastly, the logic to skip deleted files needs to be special cased
when we're faking paths like this: if we've added data for a file
in the testable engine, we should also consider that file as existing.

Test Plan:
`arc unit --everything` here, and passing our tests in
our repo over there.

Reviewers: epriestley, mareksapota

Reviewed By: epriestley

CC: Korvin, aran

Differential Revision: https://secure.phabricator.com/D6841