Fixed typo in docs
Fixed bug with unicode data in email causing SQL failures
Applied fix in 6c7d2641f0aa6c944c39 to MessageLog
Explicitly open the shared connection

If we don't open it ourselves, the Django mail code happily closes it...
every single email, which defeats the whole purpose of keeping one
connection. Make sure the connection stays persistent the entire time we are
using it.

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Huge performance gains for the message generator

We were doing three to five count queries for every single message yielded
for no real purpose at all. Tell the database to give us the highest
priority and oldest message without any of the count craziness and save
ourselves a whole bunch of hassle. When we finally get to a point where
there are no messages left, bail just as we did before.

Here are some before and after observations, also including a fix not to
double-save the message log object:

BEFORE (178 message run):
      1 SELECT COUNT(*) FROM `mailer_message` WHERE `mailer_message`.`priority` < '?'
    178 DELETE FROM `mailer_message` WHERE `id` IN (?)
    178 INSERT INTO `mailer_messagelog` ...
    178 SELECT (?) AS `a` FROM `mailer_messagelog` WHERE `mailer_messagelog`.`id` = ?  LIMIT ?
    178 SELECT <fields> FROM `mailer_message` WHERE `mailer_message`.`priority` = '?'  ORDER BY `mailer_message`.`when_added` ASC LIMIT ?
    178 UPDATE `mailer_messagelog` SET `message_data` = <data> WHERE `mailer_messagelog`.`id` = ?
    371 SELECT COUNT(*) FROM `mailer_message` WHERE `mailer_message`.`priority` = '?'
    534 commit

AFTER (238 message run):
    238 DELETE FROM `mailer_message` WHERE `id` IN (?)
    238 INSERT INTO `mailer_messagelog` ...
    239 SELECT <fields> FROM `mailer_message` WHERE `mailer_message`.`priority` < '?'  ORDER BY `mailer_message`.`priority` DESC, `mailer_message`.`when_added` ASC LIMIT ?
    476 commit

Using ratios rather than hard numbers because the sending runs were slightly
different, we have ~7.1 queries/message before vs ~3.0 queries/message
after. Additionally we have one fewer commit per message as well. This will
be reduced to one in a future commit.

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Commit once per sent or deferred message

This makes a lot of sense anyway as the log entry and message deletion or
deferral should be treated as one atomic unit. It also has the benefit of
reducing the number of times we commit in the database, which can have a
huge impact on bulk sending speed.

For each sent message, we are now down to four database actions: SELECT the
message to send, INSERT the success log entry, DELETE the original message,
and COMMIT the whole result.

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Remove now-defunct message priority methods

Should be no need to use these anywhere anymore; one can always do a filter
if they truly need this granularity on the messages in the queue.

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Defer messages where the email content is None

I mean, no one would ever think of adding or editing an email message in the
admin as it stands right now, right?

Wait, captain. When someone does that it prevents all email queued up from
ever sending. That does not sound ideal whatsoever. Skip these messages so
the rest of the email can flow freely.

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Make admin a bit more functional

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Add gitignore file for pyc

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Add a mailer report view

This is useful for support staff, administrators, developers, etc. to get a
quick heads-up as to how the mailer is working. Obviously it can't answer
every question, but it should make it obvious if the mailer is failing to
send messages or is falling behind.

To enable in your project, do the normal include('mailer.urls') trick in
your root urls mapping to somewhere sane like r"^mailer/". The view itself
is protected by a permission_required decorator; to view it a user must have
the "mailer.add_message" permission.

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Use double quotes only for consistency

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Add ability to get arbitrary percentiles of delay values

A nice little hack that lets us pass values from the template to get data we
might want there. This will allow anyone to get as much or as few values as
they might need. There is also a coercion of the '_' character into a '.'
so percentiles such as '99.9' can be obtained.

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Base logging level on verbosity setting

Whenever you call a management command, you can specify a verbosity. Use
this to determine what level to log at so cron jobs can be silent most of
the time, but will still spew output when necessary. This also moves the
default logging level from debug to info, but the old loggers can be seen by
passing '-v 2' to the send_mail and retry_deferred commands.

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Add some comments for percentile class

Explaining why we have this thing anyway, and why it only declares one
special Python method.

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Fix priority inversion in generator function

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Add some info about helpful indexes

Signed-off-by: Dan McGee <dpmcgee@gmail.com>
Merge remote branch 'toofishes/master'

Conflicts:
	mailer/engine.py
Bumped version to 0.2.0
Added a setting to configure the locfile name. This enables testing with --multiprocess