Package: logjam
logjam 0.0.4Released: 14-May-2015Logjam handles the (relatively) simple problem of compressing and archiving
ISO8601 logfiles. It works like this:Write all your logs into hourly files with an ISO8601 in their filenames.Runlogjam-compressandlogjam-uploadon your log directories,
either via cron or as a daemon.Your completed logfiles will automatically be compressed and uploaded to S3.How to use itthe logfile formatYour logfiles need two things:They must contain anUTC ISO8601 timestamp. Inhaproxy-20130704T0000Z-us-west-2-i-ae23fega.log, for instance,
the timestamp is20130704T0000Z.They need to be written hourly or more frequent than hourly. Not daily.If you usersyslogorsyslog-ng, then chances are you already use hourly
files. If not, they’re very easy to configure.logjam-compressSample entry in/etc/cron.d/:10 * * * * * root logjam-compress --once /var/log/my-log-dir/Sample command to put in anupstartconfig file orrunitrun script:logjam-compress /var/log/my-log-dirlogjam-uploadSample entry in/etc/cron.d/:10 * * * * * root logjam-upload --once /var/log/my-log-dir/archive/ s3://YOUR_BUCKET/{prefix}/{year}/{month}{/{day}/{filename}Sample command to put in anupstartconfig file orrunitrun script:logjam-upload /var/log/my-log-dir/archive/ s3://YOUR_BUCKET/{prefix}/{year}/{month}{/{day}/{filename}A note on authenticationlogjam-uploadlooks for the standard boto
environment variables$AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY, plus$AWS_DEFAULT_REGIONto figure out which S3 region to use, and what creds
to use when connecting.If those variables are not present, and you happen to be runninglogjam-uploadfrom an instance with an IAM role,logjam-uploadwill parse its AWS credential from that, and connect to the local S3
region unless told otherwise with$AWS_DEFAULT_REGION.What you need to get startedJust boto, and a bucket in S3.Why is this useful?You may be right to think that don’t need this, because if you have any
significant amount of logs, you’re going to want some sort of online log
aggregation system, such as Logstash.And if you have a really significant amount of logs, you’re going to want
a really robust, distributed system for collectingandstoring logs, such as
Scribe.If you’re big enough to need the latter, then this is not the tool for you.But, if you’re small enough that don’t want to maintain your own distributed
system for storing logs, then you have two choices:Implement the online log aggregation solution, and then deal with log
persistence, probably by setting up some sort S3 output to run on
your log aggregation server. This will work, although you will have a
SPOF where you can lose all your logs for a given time period.Implement a log persistence solution whose primary machinism is
decoupled from your log aggregation solution.In the second case, you no longer have a SPOF that can lose all your logs for a
given time, although when you lose an individual server, youwill lose its logs
from its last, partial hour.I’ve been very happy with the second case, and indeed when I have to
choose which to have first, I always choose persistence over
aggregation. Unfortunately, I’m always writing code to take care of the
persistence – i.e. of compressing and uploading logfiles. So, finally,
here’s a small, open source tool for it.Running testsUnit tests run with:python setup.py testIntegration tests run with:export SENTRY_DSN=”https://SOME SENTRY DSN”
python tests/integration/test_all.pyOr run them all with:./test_all.sh