Added ldap import controller
Minor, fix an issue where fail to extract the author PHID from an edge transaction.
Don't fail in Diffusion if .gitmodules is missing

Summary:
See T1448. If this file isn't present, just move on instead of failing, since it's a (sort of) legitimate repository state.

Also fix some silliness a little later that got introduced in refactoring, I think.

Test Plan: Added an external to my test repo and removed ".gitmodules". Verified that the directory is now viewable after this patch.

Reviewers: btrahan, davidreuss, jungejason

Reviewed By: davidreuss

CC: aran

Maniphest Tasks: T1448

Differential Revision: https://secure.phabricator.com/D2922
Phame - add an "Everyone but Me" view and make published posts truly accessible to the whole world.

Summary:
accessibility covers not only a given post but also the various "published" views.

to keep the code relative clean, this diff also splits up the post list controller logic quite a bit. this also feels like good preparation for some other work around introducing "blogs" which are collections of published posts from bloggers with some fancy features around that.

Test Plan: clicked around various parts of the Phame application as a logged in user, a logged in user with no personal posts, and without any user logged in at all. various views all seemed reasonable.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T1373

Differential Revision: https://secure.phabricator.com/D2898
Allow PhabricatorGlobalLock to block

Summary: See D2924.

Test Plan: Ran locks with blocking timeouts.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D2925
Document event --trace added by D2918

Summary: Also fix a doc typo.

Test Plan: Read.

Reviewers: btrahan, jungejason

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1092

Differential Revision: https://secure.phabricator.com/D2919
Improve error message for error 2006

Summary:
See discussion here:

https://secure.phabricator.com/chatlog/channel/%23phabricator/?at=21186

Basically, MySQL usually raises a good error if we exceed "max_allowed_packet":

  EXCEPTION: (AphrontQueryException) #1153: Got a packet bigger than 'max_allowed_packet' bytes

But sometimes it gives us a #2006 instead. This is documented, at least:

>"With some clients, you may also get a Lost connection to MySQL server during query error if the communication packet is too large."

  http://dev.mysql.com/doc/refman//5.5/en/packet-too-large.html

Try to improve the error message to point at this as a possible explanation.

Test Plan: Faked an error, had it throw, read exception message. See also chatlog.

Reviewers: btrahan, skrul

Reviewed By: skrul

CC: aran

Differential Revision: https://secure.phabricator.com/D2923
Add an Aphlict CLI client

Summary: Depends on D2926. Adds a simple CLI client for Aphlict to make it easier to debug stuff.

Test Plan: Ran client, saw debug messages.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D2927
Add a differential.abandon Conduit method.

Summary:
Brazenly copied from differential.close. Ulterior motive
is to be able to automate discarding of useless commits from
arcanist testing.

Test Plan:
Called method in phabricator sandbox. Revisions were
successfully abandoned.

Reviewers: epriestley

Reviewed By: epriestley

CC: nh, aran, Korvin

Differential Revision: https://secure.phabricator.com/D2928
Multi-line highlighting in Diffusion.

Summary: Currently, Diffusion supports highlighting of line ranges passed in the URI. It would be helpful to be able to highlight multiple line ranges.

Test Plan: Accessed directly via URL in my sandbox. Seems to work. I'm not sure what other components use this functionality, but this change should be backwards compatible.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D2921
Allow configuration of the default monospaced font style

Summary: This is a fairly contentious default that we can easily move to configuration.

Test Plan: Changed the default, changed my user setting, reverted my user setting, verified the "settings" page.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D2935
Simplify build file from data-or-hash code

Summary:
We have a bit more copy-paste than we need, consolidate a bit.

(Also switch Mercurial to download git diffs, which it handles well; we use them in "arc patch".)

Test Plan:
  - Downloaded a raw diff from Differential.
  - Downloaded a raw change from Diffusion.
  - Downloaded a raw file from Diffusion.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D2942
Update GitHub oauth docs

Summary: GitHub moved these pages around and made it easier to find your application list.

Test Plan: Read docs, verified links are correct.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D2941
Add a generic object for unit tests

Summary:
A later diff adds unit tests against edges, but we need real objects to connect with edges. Add some trivial objects to the Harbormaster database to compliment the similar HarbormasterScratchTable.
On its own, this does nothing interesting.

Test Plan: Built unit tests on this in a followup.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1162

Differential Revision: https://secure.phabricator.com/D2937
Store forced connections in the Lisk connection cache

Summary:
In unit tests which use fixtures, we open transactions on every connection we establish. However, since we don't track connections that are established with "$force_new" (currently, only GlobalLock connections) we never close these transactions normally.

Instead of not tracking these connections, track them using unique keys so we'll never get a cache hit on them.

Test Plan: Built unit tests on top of this, had them stop dying from unclosed transactions.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1162

Differential Revision: https://secure.phabricator.com/D2938
Namespace GlobalLocks to storage namespaces

Summary:
Currently, multiple unit tests that acquire global locks will interfere with each other. Namespace the locks so they don't.

(Possibly we should also rename this to PhabricatorStorageNamespaceLock or something since it's not really global any more, but that's kind of unwieldy...)

Test Plan: Acquired locks with --trace and verified they were namespaced properly.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1162

Differential Revision: https://secure.phabricator.com/D2939
Allow edges to be configured to prevent cycles

Summary:
Certain types of things we should be storing in edges (notably, Task X depends on Task Y) should always be acyclic. Allow `PhabricatorEdgeEditor` to enforce this, since we can't correctly enforce it outside of the editor without being vulnerable to races.

Each edge type can be marked acyclic. If an edge type is acyclic, we perform additional steps when writing new edges of that type:

  - We acquire a global lock on the edge type before performing any reads or writes. This ensures we can't produce a cycle as a result of a race where two edits add edges which independently do not produce a cycle, but do produce a cycle when combined.
  - After performing writes but before committing transactions, we load the edge graph for each acyclic type and verify that it is, in fact, acyclic. If we detect cycles, we abort the edit.
  - When we're done, we release the edge type locks.

This is a relatively high-complexity change, but gives us a simple way to flag an edge type as acyclic in a robust way.

Test Plan: Ran unit tests.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1162

Differential Revision: https://secure.phabricator.com/D2940
Limit maximum size of Owners package queries

Summary:
Currently, a change may affect a very large number of paths. When we run the OwnersWorker on it, we'll execute a query which looks up packages for the paths. This may exceed "max_allowed_packet". Instead, break the list of paths into smaller chunks.

This is mostly to unblock r4nt / llvm, I'm going to add a more finessed approach to array_chunk() shortly.

Test Plan: Ran `reparse.php --owners` on a revision which affected packages, verified results are the same before and after the change. Set chunk size to 1, verified query results aggregated properly.

Reviewers: btrahan, jungejason, nh

Reviewed By: jungejason

CC: aran

Differential Revision: https://secure.phabricator.com/D2943
Add a generic multistep Markup cache

Summary:
The immediate issue this addresses is T1366, adding a rendering cache to Phriction. For wiki pages with code blocks especially, rerendering them each time is expensive.

The broader issue is that out markup caches aren't very good right now. They have three major problems:

**Problem 1: the data is stored in the wrong place.** We currently store remarkup caches on objects. This means we're always loading it and passing it around even when we don't need it, can't genericize cache management code (e.g., have one simple script to drop/GC caches), need to update authoritative rows to clear caches, and can't genericize rendering code since each object is different.

To solve this, I created a dedicated cache database that I plan to move all markup caches to use.

**Problem 2: time-variant rules break when cached.** Some rules like `**bold**` are time-invariant and always produce the same output, but some rules like `{Tnnn}` and `@username` are variant and may render differently (because a task was closed or a user is on vacation). Currently, we cache the raw output, so these time-variant rules get locked at whatever values they had when they were first rendered. This is the main reason Phriction doesn't have a cache right now -- I wanted `{Tnnn}` rules to reflect open/closed tasks.

To solve this, I split markup into a "preprocessing" phase (which does all the parsing and evaluates all time-invariant rules) and a "postprocessing" phase (which evaluates time-variant rules only). The preprocessing phase is most of the expense (and, notably, includes syntax highlighting) so this is nearly as good as caching the final output. I did most of the work here in D737 / D738, but we never moved to use it in Phabricator -- we currently just do the two operations serially in all cases.

This diff splits them apart and caches the output of preprocessing only, so we benefit from caching but also get accurate time-variant rendering.

**Problem 3: cache access isn't batched/pipelined optimally.** When we're rendering a list of markup blocks, we should be able to batch datafetching better than we do. D738 helped with this (fetching is batched within a single hunk of markup) and this improves batching on cache access. We could still do better here, but this is at least a step forward.

Also fixes a bug with generating a link in the Phriction history interface ($uri gets clobbered).

I'm using PHP serialization instead of JSON serialization because Remarkup does some stuff with non-ascii characters that might not survive JSON.

Test Plan:
  - Created a Phriction document and verified that previews don't go to cache (no rows appear in the cache table).
  - Verified that published documents come out of cache.
  - Verified that caches generate/regenerate correctly, time-variant rules render properly and old documents hit the right caches.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1366

Differential Revision: https://secure.phabricator.com/D2945
Fix symbol handling in symbol query and IRC "Where is x?" handler

Summary: If a symbol's project has no linked repository, we currently explode. Instead, decline to generate a URI and fall back gracefully.

Test Plan: https://secure.phabricator.com/chatlog/channel/%23phabricator/?at=22345

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T1465

Differential Revision: https://secure.phabricator.com/D2948