Script to delete the extra EASDevices for an account.

Summary:
Certain EASAccounts have more than one pair of EASDevices due to reauth. I plan to roll out a
real fix for the issue shortly (updated: revision D727).
To back-fix, prune the superseded EASDevice pairs for these accounts.

Test Plan:
Tested on a prod replica.

Planning to run online in prod -
There are 6 affected accounts, the script takes ~5minutes to run (due to the EASUid deletes).

Reviewers: emfree

Reviewed By: emfree

Differential Revision: https://review.inboxapp.com/D726
Fix events IntegrityError for Esper and other accounts

Summary:
Hey,

This one was super annoying. I was expecting some crazy concurrency bug with Kavya's dual-sync but in the end the bug was much more lame. EAS is actually sending us some duplicate events -- my guess is that in some cases editing an event creates a duplicate event with the exact same `UID` but different `ServerIds`. Madness.

My solution is to assume the event with the highest ServerId is the most recent one. Instead of trying to dedupe the list of events we get from EAS I chose to `flush()` the session whenever we add a new event. This solves the problem because in the case where we get a duplicate event it's considered as an update of the older event which makes the code work.

Test Plan: Tested on prod ;) More seriously, this fixed Esper's test account.

Reviewers: emfree, kav-ya

Reviewed By: kav-ya

Subscribers: spang

Projects: #eas, #sync-engine

Maniphest Tasks: T556

Differential Revision: https://review.inboxapp.com/D731
Bump version in setup.py
Remove unnecessary chmod commands from setup.sh.

Summary:
Commit e540d08add1161e301f563b12cc8f182a2fed0b0 removes `cp` commands,
which are followed by `chmod 644` commands (introduced in
a84e8be0c84b62ad099dd0ba38a352a9795b103d as a first fix to non-standard
umasks), in favor off `install -m0644` commands to copy configuration
files from the host to the guest. This makes `chmod 644` commands
useless.

Test Plan:
Create a brand new virtual machine starting with vagrant up and make sure
/etc/inboxapp/config.json and /etc/inboxapp/secrets.yml are world-readable.

Reviewers: dlitz

Reviewed By: dlitz

Differential Revision: https://review.inboxapp.com/D663
Add Redis its dependencies and configuration.

Summary:
Modify setup.sh to install and configure redis. setup.sh downloads
redis-2.8.17, which is the same available through AWS (Debian/Ubuntu's version
is far too old), compiles and installs it (through stow), configures it using
the utils/install_server.sh redis.io distributes.

Add Redis configuration parameters to etc/config-dev.json (nothing for
etc/config-test.json yet).

Test Plan:
Bring up a new virtual machine and make sure Redis is running by connecting
through redis-cli. Fire up the Python console and run the following piece of
code to make sure the redis pip package works correctly:

```
from redis import StrictRedis

client = StrictRedis(host='localhost')
print client.ping()
```

Reviewers: dlitz, kav-ya

Reviewed By: kav-ya

Subscribers: emfree

Differential Revision: https://review.inboxapp.com/D730
Fix ES datetime field parse error.
Support for ES nested objects, basic nested queries.
Note: does not support cross parent/child nested relations.

Plus, test search response returns complete API repr.
Fix exception in break_to_interpreter when running with a PID < 1024 (i.e. Docker)

Test Plan:
Check that the service no longer fails when running under
Docker.

Reviewers: khamidou

Reviewed By: khamidou

Differential Revision: https://review.inboxapp.com/D733
Support Docker-based development environment

Summary:
- Merge Dockerfile.base -> Dockerfile
- "docker build ." now works.
- "fig build && fig up" now works.

The full command I use to build everything from clean sources is:

    sudo git clean -dfx && chmod -R u=rwX,g=rX,o=rX . && git-set-mtimes && fig build && fig up -d && fig logs

git-set-mtimes ensures that we don't build the requirements more often
than we have to.  It comes from here:

    https://raw.githubusercontent.com/docker-library/official-images/0bb1458242ca42b1ff117da9b7aed076c68688fc/bashbrew/git-set-mtimes

Test Plan:
    fig build && fig run inbox tox

(Note: Some tests seem to fail right now for some reason, but this
appears to be unrelated to this change.)

Reviewers: colhom, spang

Reviewed By: spang

Differential Revision: https://review.inboxapp.com/D738
config: Support loading config from INBOX_CFG_PATH environment variable

Summary:
This makes it easier to change the configuration from within a
separate development environment.

Test Plan: None

Reviewers: colhom, spang

Reviewed By: spang

Differential Revision: https://review.inboxapp.com/D739
blockhead
Also catch ValueError in message parsing.

Summary: This will let us diagnose T612.

Test Plan: None.

Reviewers: khamidou

Differential Revision: https://review.inboxapp.com/D751
Remove RETRY_FAIL_CLASSES in mailsync.

For better or for worse, we really have to keep retrying on things like
transient IntegrityErrors due to race condition, or strange errors raised in
message parsing.
better names
Add draft infrastructure to monitor the sync status.

Summary:
This commit introduces a draft infrastructure to monitor the sync
status. When an account monitor starts a new folder engine, the folder
engine ctor instantiates an object of type SyncStatus and eagerly signal
its presence to the world by publishing a timestamp on Redis with a key
composed of the account and folder identifier (this makes searches for
all folders of a given account really simple with a "regex" like
"(<account idenfitier>, *)").
As of now, folder engines keeps signaling their "state" every 60 seconds
(this is a configurable parameter at SyncStatus instantiation) and
eagerly do so when the "state" changes (e.g., on startup or state
transition).

NOTE: reducing the interval between subsequent signal operations (e.g.,
from 60 to 10 seconds) may not work correctly since the folder engines
are sleeping for "poll_frequency" and might also idle. This should not
be a huge limitation since having up to date monitoring within a minute
should be totally reasonable.

WARNING: This commit also add a very primitive script to flush ALL the
keys of ALL the dbs handled by the Redis instance pointed by the
REDIS_HOSTNAME, REDIS_PORT contained in /etc/inboxapp/config.json.

Add Redis configuration parameters to etc/config-test.json.

NOTE: this revision also revert a few commits I made on when I was "playing"
with sync status.

Test Plan:
Nothing really. You can see very basic information by accessing Redis from its
CLI. Try redis-cli from within the virtual machine and then scan 0 while
accounts are sycing.

Reviewers: kav-ya, emfree

Reviewed By: emfree

Subscribers: spang

Differential Revision: https://review.inboxapp.com/D746
docker: Support for running non-root cron/anacron jobs installed to /srv/cron.{hourly,daily,weekly,monthly}/

Test Plan: Ran inbox; ran 'ps'; saw that /usr/sbin/cron was running

Reviewers: spang

Subscribers: colhom

Differential Revision: https://review.inboxapp.com/D749
fail gently when checking git commit
Support for multiple Google Calendars + a bunch of fixes

Summary: This diff adds support for syncing and syncback of all the calendars available to an user. I also fixed some "bugs" with the syncback code -- notably, it's now possible to add participants to an event.

Test Plan: Tested manually.

Reviewers: emfree, kav-ya

Reviewed By: kav-ya

Subscribers: spang

Projects: #sync-engine

Maniphest Tasks: T613

Differential Revision: https://review.inboxapp.com/D745
Silence some flake8 errors

Summary:
I find some flake8 errors very annoying, particularly those about hanging indent. This code triggers a flake8 error:
```
very_long_function_name(
      param1, param2, param3)
```
While this one does not:
```
very_long_function_(
param1, param2, param3)
```

I personally find the former way more readable. This diff disables this warning.

Test Plan: Manual tests.

Reviewers: emfree, kav-ya

Reviewed By: kav-ya

Differential Revision: https://review.inboxapp.com/D759
Add support for excluding object types from delta sync API.

Summary:
This allows clients who don't care about, say, calendar events to limit
the changeset they get from the /delta and /delta/streaming endpoints, for
better performance.

Test Plan: Unit tests updated.

Reviewers: khamidou

Subscribers: bengotow

Differential Revision: https://review.inboxapp.com/D757