Support searching for users to find their LDAP entry

Summary:
 - the current LDAP auth flow expects a DN to look like
   cn=ossareh,ou=Users,dc=example,dc=com
 - however many LDAP setups have their dn look something like
   cn=Mike Ossareh,ou=Users,dc=example,dc=com

Test Plan:
 Test if logins work with a LDAP setup which has cn=Full Name
 instead of cn=username.

 To test you should ensure you set the properties needed to
 trigger the search before login as detailed in conf/default.conf.php

Reviewers: epriestley

CC: mbeck, aran, Korvin

Differential Revision: https://secure.phabricator.com/D3072
Update documentation for D3045

Summary: Some minor updates for changes in D3045 and clarity.

Test Plan: Read documentation.

Reviewers: dschleimer, bos, btrahan, vrana

Reviewed By: dschleimer

CC: aran, cbiesinger

Maniphest Tasks: T1507, T1546

Differential Revision: https://secure.phabricator.com/D3046
Move qsprintf() test cases from libphutil to Phabricator

Summary: Also move the other tests up so they'll trigger when this stuff is touched.

Test Plan: liberate

Reviewers: nh, btrahan, vrana

Reviewed By: nh

CC: aran

Maniphest Tasks: T1283

Differential Revision: https://secure.phabricator.com/D3074
Extend LiskMigrationIterator from PhutilBufferedIterator

Summary: PhutilBufferedIterator now implements all the nonspecific logic here.

Test Plan:
Created a test script like this:

  $iterator = new LiskMigrationIterator(new DifferentialRevision());
  $iterator->setPageSize(3);

  foreach ($iterator as $key => $rev) {
    echo "{$key}: ".$rev->getID()."\n";
  }

Ran it and verified sensible iteration results.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1562

Differential Revision: https://secure.phabricator.com/D3076
Add PhutilFactsUpateIterator

Summary: This iterator processes objects that have been updated.

Test Plan:
Ran this test script:

  $cursor = null;
  $table  = new DifferentialRevision();
  while (true) {
    $iterator = new PhabricatorFactsUpdateIterator($table, $cursor);
    foreach ($iterator as $new_cursor => $update) {
      echo "{$new_cursor} => D".$update->getID()."\n";
      $cursor = $new_cursor;
    }
    echo "Zzz...\n";
    sleep(5);
  }

Verified it iterated over every object and then stopped. Made a comment on a differenial revision, verified it iterated over the object after 15 seconds.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1562

Differential Revision: https://secure.phabricator.com/D3077
Fix an overlap between two elements of the Image form input.

Summary:
The filename field and the checkbox to select the default image were
overlapping in Firefox on Linux on both the Project Edit page and the
Profile Edit page.

Test Plan: Looked at both of the pages and saw that they rendered better.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3079
Load contents of shielded files with lots of changes

Summary:
We need `$this->old` and `$this->new` in `renderShield()`.

Broken since D2358.

Test Plan: Loaded contents of file with lots of changes.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3080
Merge pull request #166 from CodeBlock/master

D3079
Fix typo in comment
Prevent the ability to scrape for valid usernames

 - return the same error message when either bind or the username search
   fails to find a user
 - config variables should use hypen and not underscore
Add a checkbox to allow administrators to re-send the welcome email to users.

Summary:
See title - This simply adds a checkbox to the "Edit User" page in the
admin view, to allow an administrator to re-send the "Welcome to Phabricator"
email.

Test Plan:
Sent myself another welcome email using the checkbox.
Created a new user using the admin panel, to make sure emails still get
sent for new users.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T1524

Differential Revision: https://secure.phabricator.com/D3081
Merge pull request #167 from CodeBlock/master

D3081
Merge pull request #168 from ossareh/support_ldap_search_for_users

Support ldap search for users
Allow specifying against commit in DiffusionRawDiffQuery

Summary: I will need this for tracking line number in Blame previous revision.

Test Plan:
  $ hg diff --rev 0:1
  $ svn diff -r 64382:64383 README

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3083
Provide user option for disabling diffusion symbol cross-references

Summary: Some people don't like these, so they should be able to turn them off.

Test Plan:
Toggled the setting on and off; loaded a page in diffusion and differential
that should have symbol cross-references, and saw that they weren't linked
when I had the setting disabled. I also checked that the symbols are still
linked when the setting hasn't been touched.

Reviewers: epriestley, vrana

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3069
Add a basic "fact" application

Summary:
Basic "Fact" application with some storage, part of a daemon, and a control binary.

= Goals =

The general idea is that we have various statistics we'd like to compute, like the frequency of image macros, reviewer responsiveness, task close rates, etc. Computing these on page load is expensive and messy. By building an ETL pipeline and running it in a daemon, we can precompute statistics and just pull them out of "stats" tables.

One way to do this is just to completely hard-code everything, e.g. have a daemon that runs every hour which issues a big-ass query and dumps results into a table per-fact or per fact-group. But this has a bunch of drawbacks: adding new stuff to the pipeline is a pain, various fact aggregators can't share much code, updates are slow and expensive, we can never build generic graphs on top of it, etc.

I'm hoping to build an ETL pipeline which is generic enough that we can use it for most things we're interested in without needing schema changes, and so that installs can use it also without needing schema changes, while still being specific enough that it's fast and we can build useful stuff on top of it. I'm not sure if this will actually work, but it would be cool if it does so I'm starting pretty generally and we'll see how far I get. I haven't built this exact sort of thing before so I might be way off.

I'm basing the whole thing on analyzing entire objects, not analyzing changes to objects. So each part of the pipeline is handed an object and told "analyze this", not handed a change. It pretty much deletes all the old data about that thing and then writes new data. I think this is simpler to implement and understand, and it protects us from all sorts of weird issues where we end up with some kind of garbage in the DB and have to wipe the whole thing.

= Facts =

The general idea is that we extract "facts" out of objects, and then the various view interfaces just report those facts. This change has on type of fact, a "raw fact", which is directly derived from an object. These facts are concerete and relate specifically to the object they are derived from. Some examples of such facts might be:

  D123 has 9 comments.
  D123 uses macro "psyduck" 15 times.
  D123 adds 35 lines.
  D123 has 5 files.
  D123 has 1 object.
  D123 has 1 object of type "DREV".
  D123 was created at epoch timestamp 89812351235.
  D123 was accepted by @alincoln at epoch timestamp 8397981839.

The fact storage looks like this:

  <factType, objectPHID, objectA, valueX, valueY, epoch>

Currently, we supprot one optional secondary key (like a user PHID or macro PHID), two optional integer values, and an optional timestamp. We might add more later. Each fact type can use these fields if it wants. Some facts use them, others don't. For instance, this diff adds a "N:*" fact, which is just the count of total objects in the system. These facts just look like:

  <"N:*", "PHID-xxxx-yyyy", ...>

...where all other fields are ignored. But some of the more complex facts might look like:

  <"DREV:accept", "PHID-DREV-xxxx", "PHID-USER-yyyy", ..., ..., nnnn> # User 'yyyy' accepted at epoch 'nnnn'.
  <"FILE:macro", "PHID-DREV-xxxx", "PHID-MACR-yyyy", 17, ..., ...> # Object 'xxxx' uses macro 'yyyy' 17 times.

Facts have no uniqueness constraints. For @vrana's reviewer responsiveness stuff, we can insert multiple rows for each reviewer, e.g.

  <"DREV:reviewed", "PHID-DREV-xxxx", "PHID-USER-yyyy", nnnn, ..., mmmm> # User 'yyyy' reviewed revision 'xxxx' after 'nnnn' seconds at 'mmmm'.

The second value (valueY) is mostly because we need it if we sample anything (valueX = observed value, valueY = sample rate) but there might be other uses. We might need to add "objectB" at some point too -- currently we can't represent a fact like "User X used macro Y on revision Z", so it would be impossible to compute macro use rates //for a specific user// based on this schema. I think we can start here though and see how far we get.

= Aggregated Facts =

These aren't implemented yet, but the idea is that we can then take the "raw facts" and compute derived/aggregated/rollup facts based on the raw fact table. For example, the "count" fact can be aggregated to arrive at a count of all objects in the system. This stuff will live in a separate table which does have uniqueness constraints, and come in the next diff.

We might need some kind of time series facts too, not sure about that. I think most of our use cases today are covered by raw facts + aggregated facts.

Test Plan: Ran `bin/fact` commands and verified they seemed to do reasonable things.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran, majak

Maniphest Tasks: T1562

Differential Revision: https://secure.phabricator.com/D3078
Jump to correct line in Blame previous revision

Test Plan: Jumped on correct line in SVN, Git and HG repos.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3084
Add aggregated facts to the Facts application

Summary:
Some facts are aggregations of other facts. For example, we may compute how many times each macro is used in each object as a "raw fact":

  Dnnn uses macro "psyduck" 6 times.

But we want to present this data in aggregate form, e.g. "order macros by popularity". We can do this at runtime and it probably won't be too awful a query, but we can also aggregate it cheaply:

  Macro "psyduck" is used 3920 times across all objects.

...and then do a query like "select macros ordered by usage".

"Aggregate" facts support facts like this. The aggregate facts I've implemented are:

  - Count of all objects.
  - Count of objects of type X.
  - Last time facts were updated.

These clearly fit the "aggregate" facts template well. I'm not 100% sure macros do. We can use this table to answer a question like "What are the most popular macros, ordered by use?" We can also use it to answer a question like "What are the most popular macros in the last 6 months?", if we build a specific fact for that. But we can't use it to answer a question like "What are the most popular macros between times X and Y?". Maybe that's important; maybe not.

This seems like a good fit for at least some types of facts.

I'll de-magic the keys a bit in the next diff.

Test Plan: Ran the engines and got some aggregated facts about other facts.

Reviewers: vrana, btrahan

Reviewed By: vrana

CC: aran

Maniphest Tasks: T1562

Differential Revision: https://secure.phabricator.com/D3089
Add the ability to append to $PATH, for when we shell out to system binaries.

Summary:
In some cases, we shell out to things (like Pygments for syntax highlighting).
However, on cloud servers or shared web servers, those binaries aren't always
installed system-wide.

This patch allows for appending to the environment variable $PATH, to look for
other, non-default places for these binaries.

Test Plan:
* Copied the patch over to a test OpenShift instance and applied it.
* Added the path to my local copy of Pygments (pygmentize wasn't available on the system)
  into the Phabricator config.
* Refreshed a Paste page, and saw colors.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3091
Merge pull request #169 from CodeBlock/master

D3091