Legalpad - a few rough edges smoothed...

Summary:
does a few smallish things... Ref T3116

 - adds an action to "sign document", thus improving visiblity of this feature from 0 to some value more than 0
 - adds a crumb on the edit page to get back to the view page
 - warns the user on the edit page IFF signatures exist for the current version that their edits could invalidate those signatures
   - adds a "needSignatures" option to the Document Query class

Test Plan: click the new UI elements and they worked. edited a document with signatures, noted warning UI, edited anyway, noted warning UI correctly disappeared on new edit. also verified a single signature had the correct translation

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T3116

Differential Revision: https://secure.phabricator.com/D7983
Aprhont - Adding cookie-prefix, as config option, and into cookie methods

Summary: Cookie-prefix should fix phabricator instances where x.com and x.y.com have conflicting cookie names

Test Plan: Pushed branch to dev.phab.example.com, logged into phab.example.com and into dev.phab.example.com.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7979
Fix php.net link to open in a new window

Summary:
From @chad. This setup link should open in a new window so you don't lose your context in resolving setup issues.

@chad, this was the only one I could find immediately, let me know if you remember seeing others that I missed.

Test Plan: Faked an error, clicked the link, got a new tab.

Reviewers: chad, btrahan

Reviewed By: chad

CC: chad, aran

Differential Revision: https://secure.phabricator.com/D7991
Make migrations more clear in backup documentation

Summary: Not sure this would have avoided the issue, but I remember a couple of other people asking about migrations, so try to make it more clear/obvious that the backup tools are also useful for migrations. Although this is reasonably obvious when you think about it, it's not very obvious when you're trying to do a migration, and maybe making it more explicit will help.

Test Plan: Read new documentation.

Reviewers: chad, btrahan

Reviewed By: chad

CC: aran

Differential Revision: https://secure.phabricator.com/D7992
Legalpad - add a view signatures page

Summary: ...needs to add a LegalpadDocumentSignatureQuery class to get this done, which is also re-deployed everywhere we were issuing raw queries. Ref T3116.

Test Plan: viewed some signatures. Verified color and footer icons showed up how I wanted them to.

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T3116

Differential Revision: https://secure.phabricator.com/D7986
unbreak trunk - silly typo from D7986

Summary: doh

Test Plan: no more fatal on home page

Reviewers: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7993
Move repository URI normalization out of PullLocalDaemon

Summary: Ref T4327. Moves us one small step forward toward testable change parsers by separating out this unrelated logic from the PullLocal daemon. We will also probably want to run this logic so we can do remote path lookups to limit the role of Arcanist Projects in the future, which is why I made the URI type (here, only "git") a parameter rather than calling this a `GitURINormalizer` or something.

Test Plan:
  - Ran unit tests.
  - Ran `repository discover` on a correctly-configured remote repository.
  - Ran `repository discover` on an incorrectly-configured remote, got an error message.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D7981
Introduce ref cursors for repository parsing

Summary:
Ref T4327. I want to make change parsing testable; one thing which is blocking this is that the Git discovery process is still part of `PullLocal` daemon instead of being part of `DiscoveryEngine`. The unit test stuff which I want to use for change parsing relies on `DiscoveryEngine` to discover repositories during unit tests.

The major reason git discovery isn't part of `DiscoveryEngine` is that it relies on the messy "autoclose" logic, which we never implemented for Mercurial. Generally, I don't like how autoclose was implemented: it's complicated and gross and too hard to figure out and extend.

Instead, I want to do something more similar to what we do for pushes, which is cleaner overall. Basically this means remembering the old branch heads from the last time we parsed a repository, and figuring out what's new by comparing the old and new branch heads. This should give us several advantages:

  - It should be simpler to understand than the autoclose stuff, which is pretty mind-numbing, at least for me.
  - It will let us satisfy branch and tag queries cheaply (from the database) instead of having to go to the repository. We could also satisfy some ref-resolve queries from the database.
  - It should be easier to extend to Mercurial.

This implements the basics -- pretty much a table to store the cursors, which we update only for Git for now.

Test Plan:
  - Ran migration.
  - Ran `bin/repository discover X --trace --verbose` on various repositories with branches and tags, before and after modifying pushes.
  - Pushed commits to a git repo.
  - Looked at database tables.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D7982
Provide an alternate, more general "closeable" flag for commits

Summary:
Ref T4327. This provides a more general RefCursor-based way to identify closeable commits, and moves away from the messy `seenOnBranches` stuff. Basically:

  - When a closeable ref (like the branch "master") is updated, query the VCS for all commits which are ancestors of the new ref, but not ancestors of the existing closeable heads we previously knew about. This is basically all the commits which have been merged or moved onto the closeable ref.
  - Take these commits and set the "closeable" flag on the ones which don't have it yet, then queue new tasks to reprocess them.

I haven't removed the old stuff yet, but will do that shortly.

Test Plan:
  - Ran `bin/repository discover` and `bin/repository refs` on a bunch of different VCSes and VCS states. The effects seemed correct (e.g., new commits were marked as closeable).

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D7984
Remove all the multi-pass autoclose-branch separate-cache / seenOnBranches junk

Summary:
Ref T4327. Simplify the git discovery process so I can move it to the DiscoveryEngine, so I can make change parsing testable.

In particular:

  - As an optimization, we process closeable branches ("master") first, then process uncloseable branches ("epriestley-devel"). This means that in the common case we can insert a commit as closeable immediately when it is discovered, the first pass through the pipeline will get it right, and the "ref update" step will never need to do any meaningful work.
  - Commits which do not initially appear on a closeable branch, but later move to one (via merges or ref moves) will now be caught in the ref update step, have the closeable flag set, and have a message step re-queued.
  - We no longer need to do a separate discovery step on closable branches.
  - We no longer need to keep track of `seenOnBranches`.

Test Plan: Ran discovery on repositories after pushing commits, got reasonable results.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D7985
Move Git discovery into DiscoveryEngine

Summary:
Ref T4327. Consolidates and simplifies infrastructure:

  - Moves Git discovery into DiscoveryEngine.
  - Collapses a bunch of the Git and Mercurial code related to stream discovery.
  - Removes all cach code from PullLocal daemon (it's no longer called).
  - Adds basic unit tests for Git and Mercurial discovery.

Various cleanup:

  - Makes GitStream and MercurialStream extend a common base.
  - Improves performance of MercurialStream in some cases, by requiring fewer commits be output and parsed.
  - Makes mirroring exceptions easier to debug.
  - Fixes discovery of Mercurial repositories with multiple branch heads.
  - Adds some missing `pht()`.

Test Plan:
I tested this fairly throughly because I think this phase is complete:

  - Made new repositories in multiple VCSes and did full imports.
    - Particularly, I reimported Arcanist to make sure that TODO was resolved. I think it was related to the toposort stuff.
  - Pushed commits to multiple VCSes.
  - Pushed commits to a non-close branch, then pushed a merge commit. Observed commits import initially as non-close, then get flagged for close.
  - Started full daemons and resolved various minor issues that showed up in the daemon log until everything ran cleanly.
  - Basically spent about 30 minutes banging on this in every way I could think of to try to break it. I found and fixed some minor stuff, but it seems solid.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D7987
Remarkup - add underline rule

Summary: we need this for legalese. Ref T3116

Test Plan: made a legalpad document with underlines. also re-gened docs and noted underlines worked

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T3116

Differential Revision: https://secure.phabricator.com/D7996
Legalpad - fix a bug for documents with no signatures

Summary: be sure to use array() (and not null) so we don't fatal if we have no signatures

Test Plan: no more fatal when editing a signature-less document

Reviewers: epriestley

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D8000
Move more discovery responsibilities into DiscoveryEngine

Summary:
Ref T4327. This moves the last pieces of discovery responsibility out of the PullLocal daemon and into the DiscoveryEngine.

(This makes it easier to discover repositories in unit tests in the future, since we don't need to build a PullLocal daemon and can just build a DiscoveryEngine.)

Test Plan: Ran `phd debug pulllocal`. Ran `repostory discover`.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D7997
Reduce parse latency for changes pushed to hosted repositories

Summary:
Currently, we can sit in a sleep() for up to 15 seconds (or longer, in some cases), even if there's a parse request.

Try polling the DB every second to see if we should wake up instead. This might or might not produce significant improvements, but seems OK to try and see.

Also a small fix for logging branch names with `%` in them, etc.

Test Plan: Ran `phd debug pulllocal` and pushed commits, saw them parse within a second.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran, dctrwatson

Differential Revision: https://secure.phabricator.com/D7998
Remove DiffusionBranchInformation in favor of DiffusionRepositoryRef

Summary: Ref T4327. At some point these two very similar classes got introduced. Collapse `DiffusionBranchInformation` into the nearly identical `DiffusionRepositoryRef`, which enjoys slightly more generality and support.

Test Plan: Viewed branch overview and detail pages. Ran `repository refs` and `repository discover`. Grepped for removed symbols.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D8002
Merge `diffusion.commitbranchesquery` into `diffusion.branchquery`

Summary:
Ref T4327. This is general cleanup since I was in this area of the code. Primarily, the Mercurial implementation here was completely broken and wrong:

  - It returned only one branch, but a commit can be present on many branches.
  - It did not account for multiple branch heads.
  - It returned a result implying the branch head pointed at the queried commit, which is no consistent or accurate.

Simplify the amount of API we're dealing with by collapsing this method into the very similar `diffusion.branchquery` method.

Test Plan: Looked at mercurial and git repositories and commits, branch information seemed correct.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D8003
Update quickstart SQL

Summary: Ref T4327. The change parser unit tests need database fixtures, which are getting a bit slow to build. Speed them up by updating the quickstart.

Test Plan: Initialized new storage via quickstart, clicked around, everything seemed to work properly. Ran all unit tests.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4327

Differential Revision: https://secure.phabricator.com/D8004
Merge remote-tracking branch 'upstream/master' into release