Add KEYDIR to tests/config.cfg
Whoops, add missing 'then' in if statement.
Rename mail sync log to be less generic.

Summary: We have contacts sync now too.

Test Plan: ./runtests passes

Reviewers: emfree

Reviewed By: emfree

Differential Revision: https://review.inboxapp.com/D53
Move mailsync.backends.imapaccount -> mailsync.backends.imap.account.

Summary:
I don't think we should have direct submodules of mailsync.backends that
aren't actual backends. This code is clearly code that belongs to the
IMAP backend, and thus belongs in that submodule.

Test Plan: ./runtests passes

Reviewers: kav-ya

Differential Revision: https://review.inboxapp.com/D52
More linting and clarity fixes.

Summary:
Cleanups.

Test Plan: ./runtests passes

Reviewers: kav-ya

Reviewed By: kav-ya

Differential Revision: https://review.inboxapp.com/D52
Make clear-db use genericized drop_everything().
'easaccount' is a user-specific table too.
Actions supports pluggable backends.

Test Plan: Gmail actions works as before.

Reviewers: spang

Reviewed By: spang

Differential Revision: https://review.inboxapp.com/D55
Fix actions unit tests.

Test Plan: I ran them.

Reviewers: kav-ya

Differential Revision: https://review.inboxapp.com/D56
Replace bad account.create_message calls.

Summary:
imap/__init__.py has:
from inbox.server.mailsync.backends.imap import account
and then calls account.create_message in two places, but I'm pretty sure that
should be account.create_imap_message.

A create_message function actually exists in the module namespace of
backends/imap/account.py, because it gets imported from
inbox.server.models.message. So this doesn't get caught by the linter.

Test Plan: No :(

Reviewers: kav-ya, spang

Reviewed By: spang

Differential Revision: https://review.inboxapp.com/D57
Don't use args unpacking in call to msg_create_fn.

Summary:
5e1141b837 broke EAS sync because create_eas_message wasn't passing a
namedtuple. I think it's a little clearer to have implementations of
msg_create_fn just take the raw message object and inspect its properties
themselves. This way you can also have the raw message object be any sort of
class, as opposed to a namedtuple specifically.

Test Plan: Ran unit tests; synced test Gmail account.

Reviewers: spang

Reviewed By: spang

Differential Revision: https://review.inboxapp.com/D58
Reset certain columns in eas keep_tables

Summary:
With just ./tools/clear-db (i.e. --with users not specified), we want to keep the easaccount table but reset
eas_account_sync_key=0 (i.e. get rid of all sync state info)

See matching push to inbox-util

Test Plan: Expected effect for both gmail, EAS.

Reviewers: spang

Differential Revision: https://review.inboxapp.com/D60
default->server_default in database tables
add util package to dependencies
Allow out-of-tree action submodules.

Test Plan: test_actions, test_actions_syncback work.

Reviewers: spang

Reviewed By: spang

Differential Revision: https://review.inboxapp.com/D62
Migration for table split

Summary:
ImapAccount -> Account, ImapAccount
Thread -> Thread, ImapThread,
imapaccount_id->account_id in Namespace, Contact

Migration for EAS tables not needed.

Test Plan:
Both upgrading and downgrading tested to work:
To test upgrade, a database dump from a pre-table split sync was used
To test downgrade, the tests/data/base_dump.sql was used)

Reviewers: spang, kav-ya

Reviewed By: spang

CC: mg

Differential Revision: https://review.inboxapp.com/D48
Fix 'Attribute history events accumulated' warning.

Summary:
Sometimes we see this warning:

/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py:1824:
SAWarning: Attribute history events accumulated on 1 previously clean instances
within inner-flush event handlers have been reset, and will not result in
database updates. Consider using set_committed_value() within inner-flush event
handlers to avoid this warning.

I think this is because the after_flush listener in versioned_session calls
Transaction.set_extra_attrs, which in turn updates the Transaction object's
namespace attribute. Because of the now-removed backref, this would induce an
update to the corresponding Namespace object's attribute. That's the 'attribute
history event' that the warning is talking about. (You can instrument the
after_flush listener and see that after creating revisions, session.dirty
contains a Namespace instance that it did not before.)

Anyways, we don't seem to be using the backref, so remove it.

Test Plan:
Previously the warning would trigger on contacts sync. Now it does
not. Unit tests passing.

Reviewers: spang

Reviewed By: spang

Differential Revision: https://review.inboxapp.com/D66
Don't put a ton of code in a __init__.py.

Summary:
The most visible reason I'm changing this is because it makes our
logging weird (it reports the filename and __init__ is not that
helpful). But also, it's not very idiomatic to put a lot of code in a
__init__.py, it's much more common to put code in a different file and
import whatever is intended to be public in the __init__.

Test Plan: ./runtests + start up ./inbox without explosions

Reviewers: kav-ya

Reviewed By: kav-ya

Differential Revision: https://review.inboxapp.com/D67
Implement a uid/thread-polling helper for IMAP/Gmail initial sync.

Summary:
In the case that a user has e.g. a very large inbox, our current design has the
shortcoming that, during the initial sync, it won't continue to download new
messages. This is untenable for the user experience, so we're solving it by
having a parallel greenlet during this time that keeps checking for new
UIDs/threads and adding them to the top of the initial sync uid/thread download
stack.

By doing this, we have to get rid of the "chunked download" feature...  if we
batch download messages in groups, we'll introduce a delay with how fast we
download new UIDs. It's not clear to me that batching actually wins us that
much speed anyway.

I think we're going to need to introduce IDLE support at some point for
the quickest receipt of new mail, but this is better than what we've got
for now.

Test Plan: ./runtests + a bunch of manual testing that has me thinking real hard about how to do better automatic testing.

Reviewers: mg

CC: kav-ya

Differential Revision: https://review.inboxapp.com/D51
Always spawn greenlets under log_uncaught_errors.

log_uncaught_errors doesn't propagate to child greenlets, so this meant
that we were losing errors generated by mailsync greenlets.