Improve error messages from Filesystem::readRandomBytes()

Summary: Fixes T4153. Primarily, raise a better error message on Windows. Also make the function a little stricter and a little better tested.

Test Plan: Unit tests.

Reviewers: btrahan, hach-que

Reviewed By: hach-que

CC: aran

Maniphest Tasks: T4153

Differential Revision: https://secure.phabricator.com/D7650
Improve the performance of phutil_loggable_string()

Summary:
Ref T4162.

  - Before doing anything character oriented, test if the whole string is fine to print as-is.
  - Reduce the number of calls to `string()`.

Test Plan: Saw cost drop on profiles.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4162

Differential Revision: https://secure.phabricator.com/D7656
Support b'binary string' in XHPASTNode

Summary: Documented at [[ http://www.php.net/manual/en/language.types.type-juggling.php#language.types.typecasting | Type Casting ]].

Test Plan: New unit test.

Reviewers: epriestley, vinodsr

Reviewed By: epriestley

CC: Korvin, epriestley, aran, vinodsr

Differential Revision: https://secure.phabricator.com/D7662
Fix daemons erroring on already closed `STDOUT` and `STDERR`

Summary: Fixes T4177. This is the only "good" way of not erroring on these lines, because detecting truthyness doesn't always tell you everything.

Test Plan: after changing the lines, `tail`ed the log. Didn't see new errors coming in. Changed the lines back and saw the log growing quite quickly.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Maniphest Tasks: T4177

Differential Revision: https://secure.phabricator.com/D7674
Raise an exception if LDAP search fails

Summary: Currently, we assume search will always succeed if bind succeeds, but this isn't true in the wild. We don't have a root cause yet, but blunt this error into something vaguely useful rather than passing `null` to an `array` typehint and failing later and less usefully.

Test Plan: Used `bin/auth ldap`.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7679
Add a signal handler for SIGUSR2.

Summary: For being able to notify (/waking) a sleeping daemon, a signal handler for the SIGUSR2 is added to the `DaemonOverseer`. This signal will be send to the child process which can implement a signal handler.

Test Plan: Put a PhutilDaemon (with a signal handler on SIGUSR2) to sleep for 10 minutes. Meanwhile, send a SIGURS2 to the daemon and see it being waked up.

Reviewers: epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran, richardvanvelzen

Differential Revision: https://secure.phabricator.com/D7676
Make it safe to call ExecFuture->resolveKill() after process resolution

Summary:
Fixes T4165. Currently, doing something like this is fine:

  $exec_future->resolve();
  $exec_future->resolve();

...but this isn't:

  $exec_future->resolve();
  $exec_future->resolveKill();

...and will raise garbage in the `resolveKill()` call. This is unhelpful, unexpected, and undesirable. Calling `resolve*()` multiple times is explicitly allowed, since it makes many patterns much easier to write and isn't sketchy or surprising or anything.

Check for an existing result before performing the kill.

Test Plan: Added a failing unit test and made it pass.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4165

Differential Revision: https://secure.phabricator.com/D7688
Speed up text markup for test results

Summary: Recently there are quite a few diffs hitting timeout when loaded in phabricator page. The reason is that those tests have huge amount of test results and it takes more than two minutes to finish rendering the user data part of the test results. And more than 90 percent of the time was spending on the shouldMergeBlocks calls. There is some unnecessary work in that function, especially first concatenate the lines and then split it using regex, which is a time killer.

Test Plan: Load D1071019. It has more than 15k test results and always times out before. Now it loads in about 5 seconds. Click on 'show all test results' seems to bring up all the test results in right format.

Reviewers: JoelB, lifeihuang, epriestley, #blessed_reviewers

Reviewed By: epriestley

CC: Korvin, epriestley, aran

Differential Revision: https://secure.phabricator.com/D7690
Work around broken PHP fwrite() on nonblocking pipes

Summary:
Fixes T4219. See comments. PHP returns `0` when writing to nonblocking pipes for both "everything is OK, but this pipe is blocked, so you should wait a little bit and try again" and "nothing is OK, this pipe is broken forever".

This was causing an busy loop in `ssh-exec`, where it would check if `stderr` was writable, see it was, try to write to it, get a 0-length write, interpret that as "wait a bit", wait for it to become writable (i.e., immediately!), try to write again, etc.

This was the best workaround I could come up with. I think it should be pretty safe; I'm not aware of any temporary condition or event which can make a pipe unwritable after a select says it's writable other than a write.

Test Plan: Ran `git pull` a bunch of times and `^C`'d it partway through. Previously I could leave a busy process eating a CPU on the server about 50% of the time, depending how good my timing was. I can no longer generate these processes.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4219

Differential Revision: https://secure.phabricator.com/D7748
Add color output support for supported platforms on Windows

Modifies the `disableANSI` code to not outright reject Windows
but check the `TERM` environment variable in case it supports
ANSI colors. This applies to CygWin, PowerShell and integrated
terminals such as the ones in IntelliJ IDEs.

Tested manually in PowerShell, ConEmu and `cmd.exe`.

See: <https://github.com/facebook/libphutil/pull/30>

Reviewed by: epriestley
Implement read streaming on HTTPSFutures

Summary: See D7723.

Test Plan:
  <?php

  require_once 'scripts/__init_script__.php';

  $future1 = new HTTPSFuture('https://www.google.com/');
  $future2 = new HTTPSFuture('https://www.facebook.com/');

  $wait = 0.01; // In seconds. This is very small for demo purposes, a value
                 // like "1" is probably better in practice.

  $futures = array('google' => $future1, 'facebook' => $future2);
  foreach (Futures($futures)->setUpdateInterval($wait) as $key => $future) {

    // This might be a periodic update, in which case $future will be null.
    // It might also be a future exiting, in which case $future will not be null.
    // In either case, let's read any data we can first.
    foreach ($futures as $key => $future) {
      $new_data = $future->read();
      // Throw the buffered data away after we read it.
      $future->discardBuffers();

      $len = strlen($new_data);
      echo "Got {$len} bytes from {$key}.\n";
      if (strlen($new_data)) {
        // Do something interesting with the data.
      }
    }

    // Now, check if a future exited.
    if ($future !== null) {
      // This future has exited. We should sleep for a bit and then restart it,
      // or whatever.
      echo "Future {$key} exited!\n";
    }
  }

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7724
Add updateEnv() to ExecFuture for adjusting environmental variables

Summary: Ref T4195. I have a use case where I need to tweak one variable that something else has already set with setEnv().

Test Plan: Used this in next diff.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4195

Differential Revision: https://secure.phabricator.com/D7762
Dramatically improve subprocess I/O for large buffers

Summary:
Ref T4189. When we `write()` a large buffer to an `ExecFuture`, the `fwrite()` loop currently looks like this:

  $bytes_written = fwrite($socket, $buffer);
  $buffer = substr($buffer, $bytes_written);

This is normally fine, but `substr()` is approximately `O(N)` in the size of the new string, since it has to allocate and copy it.

Instead, add `PhutilRope`, which stores a string as a list of small buffers. This allows us to remove bytes from the beginning of the string very cheaply.

In particular, this can occur when you `git push` a very large repository. If we read off the network faster than we write to `git receive-pack`, we end up with a very large internal buffer which is expensive and slow to write through.

Test Plan:
I ran this test script before and after the changes:

  <?php

  require_once 'scripts/__init_script__.php';

  $large = str_repeat('x', 1024 * 1024 * $argv[1]);

  echo "OK...\n";

  $future = new ExecFuture('cat');
  $future->write($large);
  $future->resolvex();

  echo "OK.\n";

A `32MB` write took `16s` before the change and `400ms` afterward. Generally, cost is close to `O(N)` now and was close to `O(N^2)` before, in the size of the buffer.

Reviewers: btrahan, zeeg

Reviewed By: btrahan

CC: aran, FacebookPOC

Maniphest Tasks: T4189

Differential Revision: https://secure.phabricator.com/D7768
Make PhutilChannel::waitForAny more reasonable about detecting unselectable channels

Summary:
Ref T4189. Currently, channel `getWriteSockets()` methods return only //blocked// write sockets, and `waitForAny()` degrades to a busy wait if any channels are missing read and write sockets, assuming the channel is not a socket-based channel.

This means that if you have an unblocked, writable, socket-based channel and a blocked, readable socket-based channel, we always degrade to busy wait immediately.

An example of this occurs during `git push`:

  - We have one R/W channel through SSH to the `git push` on the user's computer.
  - We have one R/W channel through ExecFuture to the `git receive-pack` on the remote host.
  - When the client finishes writing, we close the "R" half of the SSH channel. This leaves it writable, socket-based, and unblocked for most of the push.

Instead, make `getWriteSockets()` methods return //all// write sockets, even if they're not blocked, and then ignore them if `isWriteBufferEmpty()` says we don't need to wait for them.

This is generally cleaner, and fixes several issues:

  # The `ssh-exec` process no longer sits in a busy wait between the SSH read channel closing and the operation finishing.
  # The stdin/stdout/stderr log in the SSH stuff can be simplified (next diff).

Test Plan: Ran `git push` of the Wine repository, saw nothing we own spend an unreasonable amount of time doing CPU stuff. Added a bunch of logging to confirm things are working reasonably under the hood.

Reviewers: btrahan, zeeg

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4189

Differential Revision: https://secure.phabricator.com/D7769
Add a maximum read buffer size to ExecFuture

Summary:
Ref T4241. Currently, `ExecFuture` may read an arbitrarily large amount of data into memory during execution. In T4241, this arises specifically with `git-upload-pack`, which emits about as much data as the size of the repository. Currently, the subprocess emits this data far faster than we can send it over the network, so we end up with a huge in-process buffer containing the entire repository.

Instead, we want to throttle `git-upload-pack` until the network can catch up. This is the first step in letting that happen.

Test Plan: Added a unit test.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4241

Differential Revision: https://secure.phabricator.com/D7772
Use PhutilRope for Channel write buffers and make write buffer details public

Summary:
Ref T4241. Two changes:

  - Use PhutilRope instead of plain string for the Channel write buffers. This makes it more consistent, and should improve performance in some cases.
  - Expose information about buffer sizes as `public`. We need this in a future diff to be able to throttle reads when we're blocked on writes, since we have to be able to test if a write channel is blocked or not.

Test Plan: Ran `arc unit`. Ran `git` operations over SSH.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4241

Differential Revision: https://secure.phabricator.com/D7773
Add a read buffer size to PhutilChannel

Summary: Ref T4241. See D7772. Just like `ExecFuture`, allow the internal channel buffer to be limited in size so when we're plugging a fast pipe into a slow pipe we don't end up with all of the data in memory (subject to actually using these features).

Test Plan: Added unit tests.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4241

Differential Revision: https://secure.phabricator.com/D7774
Allow PhutilChannel to select only write sockets, and expose read buffer status

Summary: Ref T4241. Ref T4206. Small API changes to fix T4241 (next diff). See previous diffs for context.

Test Plan: See next revision.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T4206, T4241

Differential Revision: https://secure.phabricator.com/D7775
Make all "protected" in ExecFuture into "private"

Summary: This class is final, these are just vestigal from long, long ago.

Test Plan: Unit tests.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Differential Revision: https://secure.phabricator.com/D7779
Merge remote-tracking branch 'upstream/master'