Condense successive alter tables in migrations.
Defer initialization of Crispin connection pool so that it can be retried.

CrispinConnectionPool.__init__ calls CrispinConnectionPool._set_account_info(),
which does auth and related stuff. We were calling this in the FolderSyncEngine
constructor, which meant that the folder sync would just silently die if some
sort of error occurred. Not good.

In general we really shouldn't be doing this sort of work in any class
constructors at all.
[sync/missing folders] Create folders on the backend when not present. Required for hovermail.

Summary:
Providers don't always provide all of the folders that we are
expecting. One such provider is hovermail. In order to support such providers,
we need to create these folders on-demand. This may not be the best place to do
so, however I couldn't think of a better place (perhaps initial sync?).

Fixes T446

Test Plan: sync hovermail account, perform self-send test.

Reviewers: emfree

Reviewed By: emfree

Maniphest Tasks: T446

Differential Revision: https://review.inboxapp.com/D506
Only IDLE on the inbox folder for Gmail.

Summary:
This fixes a more than slightly mortifying regression in which we were idling
on ALL folders that we sync for Gmail. The problem with this is that it was
consuming all available crispin connections. Because the sync on the inbox
folder goes and gets a new all_mail_crispin_client to download threads, this
was causing the polling on the inbox folder to potentially block for an
unbounded amount of time. No bueno.

Test Plan:
Verify in sync logs that only the inbox folder sync idles, and that
it is not blocked.

Reviewers: spang

Reviewed By: spang

Subscribers: dlitz

Differential Revision: https://review.inboxapp.com/D510
Add bin/inbox-consistency-check debugging tool for gmail accounts

Summary:
There are still some differences that haven't been ironed out, and some stuff
isn't implemented (e.g. labels, tags, flags), but the major pieces are here,
and they should already be useful.

This also serves as an example of how to use setuptools entry_points to
make plugins.

Test Plan: manual testing

Reviewers: emfree

Reviewed By: emfree

Differential Revision: https://review.inboxapp.com/D507
Disable failing test for iCloud

Test Plan: Ran the test

Reviewers: charles

Reviewed By: charles

Differential Revision: https://review.inboxapp.com/D500
Don't refuse to schedule syncback actions for accounts in 'connerror' state.

Our ludicrously broken mismanagement of account.sync_state means that accounts
can more or less be in `connerror` for arbitrary reasons at arbitrary times. In
such cases, don't return HTTP 503 when a syncback action is scheduled.
Remove no-longer-valid test.
Fix typos when removing uids from download stack in IMAP sync.

This wasn't actually breaking anything, because we'd just end up trying to
download a UID that wasn't there. But still.
Bound thread length when doing custom threading.

Summary:
This should fix the recursion depth exceeded error we keep seeing. It's also
probably not a good idea to let threads get super long anyways.

Fixes T434.

Test Plan: Unit test added.

Reviewers: khamidou

Reviewed By: khamidou

Maniphest Tasks: T434

Differential Revision: https://review.inboxapp.com/D508
Fix bug in extracting pending UIDs from download stack.
[migration] Fix race-condition in account migration dealing with stopping / starting syncs on separate hosts.

Summary:
There is a race-condition dealing with migrating an account between
two separate hosts. This manifests itself in the following scenario:

- An account is syncing on 'Host A' (so 'sync_host' is set to 'Host A')
- The account is effectively migrated to 'Host B' (by either stopping the
  account and starting it again and it gets 'work stolen' by 'Host B', or
  potentially by simply changing the sync_host in the future)
- 'Host B' starts syncing the account right away, changing the 'sync_host'
  property on the account to be 'Host B'
- 'Host A' then tries to stop syncing the account. In this process, it checks
  to see if the account.sync_host matches it's own FQDN (assertion)
- This assertion fails, and therefore 'stopping the account' never happens.

As a consequence, this account remains in the 'self.monitors' list on 'Host A'
and during each iteration, 'Host A' realizes that it should remove the account
from it's list of monitors and repeats the process.

Furthermore, if the account is stopped, the sync_host is set to 'None' which
also does not match 'Host A' so this sync-engine doesn't get out of this loop.

Finally, if the account is started again and 'Host A' would have an opportunity
to re-claim the account, this also is not possible as the assertion prevents the
lock-file from being removed therefore preventing a new-sync from ever starting
again on this host.

The solution to this is that if the sync is set to be stopped and the sync_host
matches the FQDN the same code runs as before. If the sync_host doesn't match
the FQDN, then 'Host A' in the above scenario should log a warning and then
stop syncing the account, kill all monitors and remove the local lock. It
should not set the state to 'stopped' for the scenario when the account has
been migrated to a different host.

Note that account migration still has some potential issues and special attention
needs to be taken for when this occurs. Especially in regards to oustanding
sync-back actions. However, in the situation where the account is effectivly
in a steady-state, this change should allow the account to be migrated between
sync-engines.

Test Plan: Review the code.

Reviewers: emfree

Reviewed By: emfree

Differential Revision: https://review.inboxapp.com/D513
Deletes messages which aren't associated to an UID.

Summary: Fixes T464.

Test Plan: Synced an account, created a draft. Sent it and checked that the draft wasn't there anymore.

Reviewers: emfree

Reviewed By: emfree

Subscribers: mg

Maniphest Tasks: T464

Differential Revision: https://review.inboxapp.com/D491
broke the build -- it's not possible to modify a list and iterate it at
the same time with SQLAlchemy
Fix TypeError in cleanup_subject() when subjecct is None.
Fix NoneType error with message.references in threading algo.
Fix typo in remove_messages.
Fix KeyError in add_new_imapuids().
Log warning instead of error in Google contacts sync.

If there's a RequestError, it pretty much means that the developer hasn't
enabled the Google Contacts API. This shouldn't feed into error metrics.

Fixes T490.
Remove outdated comment.

The logic for this ended up in add_message_attrs().