some translation of email
update mail format
Delete stale fixtures with bin/storage destroy

Summary:
When a PhabricatorTestCase dies after creating storage fixtures, it leaves
those storage fixtures around.  This doesn't happen often, but when it does
happen it's a pain to cleanup.  The --unittest-fixtures option helps automate
that cleanup.

Test Plan:
Ran it with --dryrun, then for real.  Became overwhelmed with a Zen like peace
after regarding the tidiness and beauty of SHOW DATABASES.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3858
Add hotkey for hiding file tree

Summary:
In Differential, viewer can hit 'f' key to hide/show file tree on the left
side. Useful on narrow monitors.

Test Plan: Open any diff in Differential tool, hit 'f', watch file tree disappears

Reviewers: vrana, mattchoi, epriestley

Reviewed By: vrana

CC: aran, epriestley, Korvin

Differential Revision: https://secure.phabricator.com/D3844
add ids and phids to maniphest.query

Summary: requires add withTaskPHIDs to maniphest query class.

Test Plan: queried via conduit by task id and task phid -- respective success!

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2017

Differential Revision: https://secure.phabricator.com/D3863
Move task leasing to a dedicated query

Summary: This simplifies the fairly thorny logic of leasing tasks a bit. I'm planning to introduce another callsite shortly for Drydock.

Test Plan: Ran `bin/phd debug taskmaster`, observed sensible queries and correct operation.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3855
Factor out task execution and formalize permanent failures

Summary:
  - Clean up a TODO about permanent failures.
  - Clean up a TODO about failing tasks after too many retries.
  - Clean up a TODO about testing for bad leases.
  - Make the lease/retry implementation more flexible and natural.
  - Make completely bogus tasks fail permanently.
  - Make PhabricatorMetaMTAWorker use new `getWaitBeforeRetry()` (as intended), not hackily implement logic in `getRequiredLeaseTime()`.
  - Document worker hooks for failures and retries.
  - Provide coverage on everything.

Test Plan: Ran unit tests. Ran `bin/phd debug taskmaster`.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3859
Undumb the Drydock resource allocator pipeline

Summary:
This was the major goal of D3859/D3855, and to a lesser degree D3854/D3852.

As Drydock is allocating a resource, it may need to allocate other resources first. For example, if it's allocating a working copy, it may need to allocate a host first.

Currently, we have the process basically queue up the allocation (insert a task into the queue) and sleep() until it finishes. This is problematic for a bunch of reasons, but the major one is that if allocation takes more resources (host, port, machine, DNS) than you have daemons, they could all end up sleeping and waiting for some other daemon to do their work. This is really stupid. Even if you only take up some of them, you're spending slots sleeping when you could be doing useful work.

To partially get around this and make the CLI experience less dumb, there's this goofy `synchronous` flag that gets passed around everywhere and pushes the workflow through a pile of special cases. Basically the `synchronous` flag causes us to do everything in-process. But this is dumb too because we'd rather do things in parallel if we can, and we have to have a lot of special case code to make it work at all.

Get rid of all of this. Instead of sleep()ing, try to work on the tasks that need to be worked on. If another daemon grabbed them already that's fine, but in the worst case we just gracefully degrade and do everything in process. So we get the best of both worlds: if we have parallelizable tasks and free daemons, things will execute in parallel. If we have nonparallelizable tasks or no free daemons, things will execute in process.

Test Plan: Ran `drydock_control.php --trace` and saw it perform cascading allocations without sleeping or special casing.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3861
Fix an issue where PhabricatorWorkerLeaseQuery may lease different tasks than it selects

Summary:
We lock tasks by setting `leaseOwner` to a unique value, but the value is currently unique-to-the-process rather than unique-to-the-query. This means that if a process leases a task, then leases another task, both tasks will have the same `leaseOwner`. This can cause an issue where we go to select the task we just leased and get the other task instead, if we aren't careful about the select construction.

We can avoid this by being clever and making sure the select is constructed correctly, but making the `leaseOwner` unique to the query is much simpler and more foolproof. This guarantees we always select only the rows we just leased.

Also remove `PhabricatorGoodForNothingWorker` since `PhabricatorTestWorker` fills its role of allowing things to be tested, and simplify the unit tests since we don't need to be clever about avoiding this issue any more.

Test Plan: Ran unit tests.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3862
make Differential comment panel flexible width and nip a bit at other flexible width issues

Summary:
all sorts of stuff

 - made comment form width flexible
 - made margins element specific rather than part of differential-primary-pane
 - made box elements all veritically align left and right until code stuff
 - re-factored width calculaton stuff a bunch so only the code section has to suffer from max-width calculations; everything else can flex
 - made colspan 3 for rightmost table header element. this is so the "View Options" UI element ends up lining up correctly with the "Show All Lines" element just below

Test Plan: looked at revision view and changeset view and it all looked hot. note I did not test what things looked like with different word wrap values; that should still work given the re-factoring and not re-design here. also toggled haunted panel mode and it looked good.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2006

Differential Revision: https://secure.phabricator.com/D3866
robustify Differential and Maniphest mailhandlers wrt attachments

Summary:
a few things

- make the parent mailhandler class not send "blank body" error if you have attachments
- make both differential and maniphest append a list of attachments to the body if any exist
- BONUS - made the cc stuff work in Maniphest

Test Plan: I haven't actually tested this yet. :(  i need to figure out how to send a mail with an attachment from the command-line and figured I'd serve this up first.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2012

Differential Revision: https://secure.phabricator.com/D3868
Modernize the `drydock` script

Summary: Add a bin/drydock symlink and break it into workflows. Nothing too special here.

Test Plan: Ran `bin/drydock wait-for-lease`, `bin/drydock lease`, `bin/drydock help`, etc.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3867
Minor, fix an MTA issue introduced in D3859.

Prior to D3859, getRequiredLeaseTime() was called before doWork(), which had the critical but subtle side effect of populating `$this->message`. Instead, make the population explicit. This restores email functionality.

Test Plan: ran `phd debug taskmaster` and verified email was delivered

Auditors: btrahan
Make various Drydock improvements

Summary:
Tightens up a bunch of stuff:

  - In `drydock lease`, pull and print logs so the user can see what's happening.
  - Remove `DrydockAllocator`, which was a dumb class that did nothing. Move the tiny amount of logic it held directly to `DrydockLease`.
  - Move `resourceType` from worker task metadata directly to `DrydockLease`. Other things (like the web UI) can be more informative with this information available.
  - Pass leases to `allocateResource()`. We always allocate in response to a lease activation request, and the lease often has vital information. This also allows us to associate logs with leases correctly.

Test Plan: Ran `drydock lease --type host` and saw it perform a host allocation in EC2.

Reviewers: btrahan

Reviewed By: btrahan

CC: aran

Maniphest Tasks: T2015

Differential Revision: https://secure.phabricator.com/D3870
Hide Edit Maniphest Tasks in commit with disabled Maniphest

Test Plan: Viewed commit with enabled/disabled Maniphest.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3871
post feed stories to arbitrary uris via a new Worker

Summary: so folks can write applications and whatnot.

Test Plan: set feed.http-hooks to local dev instance (200) and localhost (500) in my conf. Verified succcess and retrying respectively.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T305

Differential Revision: https://secure.phabricator.com/D3874
Use getProductionURI instead of getURI for emails.

Summary: 'TASK DETAIL' links point to the non-production uri.  Our daemons run in an environment that uses different baseUrl because we can't use https locally (https is provided by our load balancers)

Test Plan: check emails generated with a non-production environment. See that the TASK DETAIL link points to production url.

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3877
Specify config for text editors

Test Plan: D3865

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Differential Revision: https://secure.phabricator.com/D3879
Delete license headers from files

Summary:
This commit doesn't change license of any file. It just makes the license implicit (inherited from LICENSE file in the root directory).

We are removing the headers for these reasons:

- It wastes space in editors, less code is visible in editor upon opening a file.
- It brings noise to diff of the first change of any file every year.
- It confuses Git file copy detection when creating small files.
- We don't have an explicit license header in other files (JS, CSS, images, documentation).
- Using license header in every file is not obligatory: http://www.apache.org/dev/apply-license.html#new.

This change is approved by Alma Chao (Lead Open Source and IP Counsel at Facebook).

Test Plan: Verified that the license survived only in LICENSE file and that it didn't modify externals.

Reviewers: epriestley, davidrecordon

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2035

Differential Revision: https://secure.phabricator.com/D3886
make is to when a user resigns from an audit they are no longer cc'd

Summary: do this by making sure to filter out those who've "resigned" from the email CC list

Test Plan: resigned from an audit and no longer got emails on updates

Reviewers: epriestley

Reviewed By: epriestley

CC: aran, Korvin

Maniphest Tasks: T2033

Differential Revision: https://secure.phabricator.com/D3890