ensure addition to .bashrc in setup.sh is idempotent

Signed-off-by: Jacob Emmert-Aronson <jre2718@berkeley.edu>
Don't error out in setup.py if directories already exist.

Signed-off-by: Jacob Emmert-Aronson <jre2718@berkeley.edu>
mysql requires user and password when accessing test database

Signed-off-by: Jacob Emmert-Aronson <jre2718@berkeley.edu>
Download messages in a thread one-at-a-time.

Summary:
Big threads containing embedded images etc. can use a TON of memory if
we try to download entire threads at a time. Shouldn't actually be much
of a perf loss to download sequentially, and it's way better than
vomiting over everything.

Someday we may want to do something fancier like precalculating message
sizes and using that to chunk.

Test Plan: Ran a sync manually. (no test coverage yet)

Reviewers: emfree

Reviewed By: emfree

Differential Revision: https://review.inboxapp.com/D123
Fix test_hooks.
Merge pull request #7 from jre21/master

Cleanups for dev environment
Add ability to attach extra data to transactions.

Summary:
The immediate reason for this is that we may want to filter message entries in
the transaction log based on folder names or attachment filenames, etc. But
that data may only appear somewhere else altogether in the log. Attaching the
data where we'll need it is much more convenient than needing to go looking for
it when we process the log.

Test Plan: Basic unit test added; sync manually run.

Reviewers: spang, mg

Reviewed By: mg

Differential Revision: https://review.inboxapp.com/D124
[SCHEMA] Add global unique `public_id` to database objects

Summary:
`public_id` is a binary column that is stored as a 128 bit unsigned integer, generated from Python's UUID module.

Although the field is stored as binary, the value on our model objects is exposed as a base-36 string (letters and numbers only). These should always be autogenerated, with the exception of the included migration.

Currently these objects have public_ids. (We can add others later if needed.)

- Account
- Block
- Contact
- Message
- Namespace
- SharedFolder
- Thread
- User
- UserSession

Test Plan: Run migration, it might take a while

Reviewers: spang, emfree

CC: kav-ya

Differential Revision: https://review.inboxapp.com/D125
Remove unused/duplicate address parsing code.
Remove PYTHONIOENCODING from setup.sh
Pool crispin clients, not IMAPClients.

Summary:
It turns out that my original abstraction here was not the right one.
Crispin's state caching essentially ties it to a single IMAP connection,
so the right thing to do is pool wrapped connections, not this mess with
passing raw IMAPClients into every single method call.

This connection pooling strategy is *definitely* not smart enough in the
long run when we have connections for many accounts open to the same
server, but it's way better than what we have now.

As a part of this refactor, we delete inbox.server.pool and move the
code in it to places that make more sense. This module was a remnant of
the days when we only supported Gmail sync.

This refactor really helps make the IMAP sync code more readable.

Test Plan: ./runtests -a

Reviewers: kav-ya

Reviewed By: kav-ya

Differential Revision: https://review.inboxapp.com/D126
Lint models.namespace.

Getting rid of old style slowly as we touch files that haven't been
touched in a while. :)
[SCHEMA] Establish an association between messages and contacts.

Summary:
We want this mostly so that we can filter messages based on the from/to/cc/bcc
fields -- constructing SQL queries against the current JSON-encoded fields is
extremely clunky.

This commit also removes the HookManager abstraction, which we probably won't
use since we're not doing synchronous hooks.

Test Plan: Unit tests updated.

Reviewers: spang, mg

Reviewed By: spang

Differential Revision: https://review.inboxapp.com/D127
Use flanker instead of manual parsing when canonicalizing adresses.
Bump alembic dep to latest.

Not fixing anything specific, but want to keep up-to-date.
Rearrange imports in previous migration.

According to mg bad things can happen if you try to do stuff outside of a
migration's upgrade() function.
Construct database queries on threads/messages from query parameters.

Summary: Currently only for exact term matching -- no regular expressions yet.

Test Plan: Unit tests to come.

Reviewers: mg, spang

Reviewed By: mg

Differential Revision: https://review.inboxapp.com/D128
Install gevent via pip instead of apt-get.

This lets us use gevent 1.0, which has more features and no external
dependencies.
First part of API (w/ migration)

Summary:
- Migration to store message `blocks` in a separate table `parts`. This change is important for future use where non-message blocks can be uploaded to a namespace. The `part` objects are attachments on messages, and inherit from the `block` foundation, which provides the appropriate hashing, IDs, and Blob storage to S3 options.

- Custom JSON Encoder: works with many of our data objects (not all yet)

- API Documentation: these are written in Markdown and live in separate category files. The Flask app handles Markdown -> HTML rendering at the `/docs` URL. CSS based on GitHub style. Table of Contents is autogenerated based on filename ordering and title metadata.

- Initial version of the API server. (See the docs for more details) The Flask server `srv.py` runs on port 5555 and most of it is broken. It uses Flask Blueprints for different URL prefixes.

- File uploads: created in proper namespace

- File downloads: smart filenames are generated based on mimetype and common extensions (inline attachments don't have filenames)

- Fix for a *huge* bug parsing messages where we replaced newlines for all binary data (not just plain text)

Test Plan: all tests pass. updated test dump included

Reviewers: spang, emfree, kav-ya

Reviewed By: spang

Differential Revision: https://review.inboxapp.com/D129
Rename Alembic migrations in the actual order

I have no idea why they don't just remove pyc files before running anything. Causes so many headaches. If you're ever having issues, try just running `find . -name '*.pyc' -delete` in your shell.