better names
Add draft infrastructure to monitor the sync status.

Summary:
This commit introduces a draft infrastructure to monitor the sync
status. When an account monitor starts a new folder engine, the folder
engine ctor instantiates an object of type SyncStatus and eagerly signal
its presence to the world by publishing a timestamp on Redis with a key
composed of the account and folder identifier (this makes searches for
all folders of a given account really simple with a "regex" like
"(<account idenfitier>, *)").
As of now, folder engines keeps signaling their "state" every 60 seconds
(this is a configurable parameter at SyncStatus instantiation) and
eagerly do so when the "state" changes (e.g., on startup or state
transition).

NOTE: reducing the interval between subsequent signal operations (e.g.,
from 60 to 10 seconds) may not work correctly since the folder engines
are sleeping for "poll_frequency" and might also idle. This should not
be a huge limitation since having up to date monitoring within a minute
should be totally reasonable.

WARNING: This commit also add a very primitive script to flush ALL the
keys of ALL the dbs handled by the Redis instance pointed by the
REDIS_HOSTNAME, REDIS_PORT contained in /etc/inboxapp/config.json.

Add Redis configuration parameters to etc/config-test.json.

NOTE: this revision also revert a few commits I made on when I was "playing"
with sync status.

Test Plan:
Nothing really. You can see very basic information by accessing Redis from its
CLI. Try redis-cli from within the virtual machine and then scan 0 while
accounts are sycing.

Reviewers: kav-ya, emfree

Reviewed By: emfree

Subscribers: spang

Differential Revision: https://review.inboxapp.com/D746
docker: Support for running non-root cron/anacron jobs installed to /srv/cron.{hourly,daily,weekly,monthly}/

Test Plan: Ran inbox; ran 'ps'; saw that /usr/sbin/cron was running

Reviewers: spang

Subscribers: colhom

Differential Revision: https://review.inboxapp.com/D749
fail gently when checking git commit
Support for multiple Google Calendars + a bunch of fixes

Summary: This diff adds support for syncing and syncback of all the calendars available to an user. I also fixed some "bugs" with the syncback code -- notably, it's now possible to add participants to an event.

Test Plan: Tested manually.

Reviewers: emfree, kav-ya

Reviewed By: kav-ya

Subscribers: spang

Projects: #sync-engine

Maniphest Tasks: T613

Differential Revision: https://review.inboxapp.com/D745
Silence some flake8 errors

Summary:
I find some flake8 errors very annoying, particularly those about hanging indent. This code triggers a flake8 error:
```
very_long_function_name(
      param1, param2, param3)
```
While this one does not:
```
very_long_function_(
param1, param2, param3)
```

I personally find the former way more readable. This diff disables this warning.

Test Plan: Manual tests.

Reviewers: emfree, kav-ya

Reviewed By: kav-ya

Differential Revision: https://review.inboxapp.com/D759
Add support for excluding object types from delta sync API.

Summary:
This allows clients who don't care about, say, calendar events to limit
the changeset they get from the /delta and /delta/streaming endpoints, for
better performance.

Test Plan: Unit tests updated.

Reviewers: khamidou

Subscribers: bengotow

Differential Revision: https://review.inboxapp.com/D757
Don't strip quoted text.

Summary:
Too unreliable for our use case without significant changes to Talon's
internals.

Test Plan: Run message-parsing unit tests.

Reviewers: spang, mg

Reviewed By: mg

Differential Revision: https://review.inboxapp.com/D761
Fix IMAP UIDVALIDITY for Gmail.

Summary:
This commit implements the resync_uids_from in GmailFolderSyncEngine.
Gmail makes our life easier here, we exploit the availability of
X-GM-MSGIDs to update the messages with their new UIDs and the folder
with the new UIDVALIDITY.

Close T601.

Test Plan:
Bring up a virtual machine, restore this
https://inboxapp.slack.com/files/emfree/F02V5ALCJ/bad_uidvalidity_dump.sql
database dump, run bin/inbox-start, and make sure the 'uidinvalid' folder gets
fixed.

Reviewers: spang, emfree

Subscribers: khamidou

Maniphest Tasks: T601

Differential Revision: https://review.inboxapp.com/D736
Temporarily proxy Gmail SMTP connections.

Investigating a spam issue.
Script to delete the extra Folders for an EASAccount.

Summary: To back-fix the prod account affected by T600.

Test Plan: Will test against a restored snapshot.

Reviewers: emfree

Reviewed By: emfree

Differential Revision: https://review.inboxapp.com/D769
Fix broken syncing tests

Summary: In an epic feat of backpedaling, I realized that the "refactors" I made to the syncing algorithm made it susceptible to a race condition. I reverted the fixes and the tests now pass.

Test Plan: Ran the tests

Reviewers: emfree

Reviewed By: emfree

Subscribers: spang

Differential Revision: https://review.inboxapp.com/D770
Check for All Mail folder when creating Gmail accounts

Summary:
This is the sync engine half of T495. UI part in progress.

Adds a Gmail-specific config check to GmailAuthHandler.
This checks for an 'All Mail' folder before creating an account.
Throws a ConfigError for the UI to handle if the folder's not found.

Test Plan:
tests/general/test_folders.py - uses a mocked out auth handler.

System test using real credentials coming separately (per spang's suggestion).

Reviewers: emfree

Reviewed By: emfree

Subscribers: kav-ya, mg, spang

Differential Revision: https://review.inboxapp.com/D765
Don't try to insert ImapUids we already have.
Add a system test to check for expected broken accounts

Summary:
This is the system test for D765 and finishes up the sync engine side
of T495 - checking for All Mail folder and failing gracefully if it's
absent.

This test specifically adds another check in `test_auth` based on a
new list of live, bad credentials in accounts.py. It's purposefully
general, so other known bad credentials could be added if we want to
test for more error cases.

There are two changes required in Jenkins to make this run:

* Add another string parameter broken_accounts to identify
  broken test accounts.
* In the build script, after the line:
  `echo "credentials = $test_accounts" > tests/system/accounts.py`
  add:
  `echo "broken_credentials = $broken_accounts" >> tests/system/accounts.py`

It looks like I can make the changes myself but I'd appreciate input
from folks more familiar with the jenkins setup.

Test Plan: If you want to test locally, ping me for some broken credentials you can use.

Reviewers: emfree

Reviewed By: emfree

Subscribers: spang

Differential Revision: https://review.inboxapp.com/D773
Update more attrs in reconcile_message().

Summary:
Like full_body, msg.sanitized_body, msg.snippet need to be
updated when reconciled too.

Test Plan: Tests pass as before.

Reviewers: emfree

Reviewed By: emfree

Differential Revision: https://review.inboxapp.com/D777
Remove unused params in api/kellogs.encode()
System tests for events creation

Summary: This is a simple smoke test to check that event creation and deletion works. I'll be interested in your thoughts on whether it's necessary to test syncback (e.g: by connecting to the google backend and querying directly).

Test Plan: Ran the tests

Reviewers: emfree, kav-ya

Reviewed By: kav-ya

Subscribers: spang

Differential Revision: https://review.inboxapp.com/D772
update docs to note that calendar events are synced back.
Also update note about syncback in the Calendars doc section