Add global ignore set to all jobs.  #86.
Ignore wayback.archive.org
Remove a rule obsoleted by one in the global set
Copy share link ignores and tracking bugs from the blogs set
Remove ignores moved to the global set
Ignore &hilit=
!igset: confirm ignore set names.

Useful in general, but particularly nice to have for the
automatically-added global ignore set.
Turn off HTTP compression to work around https://github.com/chfoo/wpull/issues/155
Send an Accept: header like Firefox 31

wpull doesn't send an Accept: header, but some sites run a WordPress plugin called "bad behavior" that requires an Accept: header when it sees a browser user agent.
Ignore del.icio.us/post
Add !pending, a command to list the pending queue.

This should probably eventually cut off at some sane limit (perhaps only
show the next ten?), but until there's another way to view the whole
list (web dashboard probably) this is the only sane way to peek inside.
Make displayed queue order match actual order.

We RPOPLPUSH from pending to working; therefore, the last item in the
pending list is what's chosen next.
Document !pending.
Replace json with yajl.

I have a completely unfounded theory that this is part of the
dashboard's process bloat.  Unfortunately, the dashboard uses the same
gems as every other Ruby program in ArchiveBot.  Consequence: swap out
every use of JSON.parse with Yajl::Parser.parse.
Revert "Replace json with yajl."

This reverts commit 63c0750f7bbea1bc7b8cdf2d402989703130ad3b.

Doesn't make a difference.
Ignore more JavaScript non-URLs
Ignore another mp3 streaming site
Introduce !ao-only mode for pipelines.  Closes #44.

In !ao-only mode, a pipeline listens only to the pending-ao queue and
its private queue.[1]

This commit also changes the bot so that all !ao jobs are put onto a
separate queue.  This lets us use the same queuing behavior that we use
for the existing queue.

[1] A pipeline in "!ao-only" mode can still do !a jobs if they're
directed specifically to it.  This is a violation of the strict reading
of "!ao-only", but pipeline-specific queues have been used in the past
to test tweaks, and I think that's a useful use case to keep around.
We'll see in whether this exception proves to be too much of a pain.
Whoops, we should still be using PhantomJS 1.9.7.  #44.
Replace LogAnalyzer with a few smaller scripts.

The cogs program is suffering from inexplicable memory leaks and/or
process bloat.  (I can't tell which one is actually happening.  Hell,
both might be occurring.)

This commit, and indeed this branch, are an attempt to rewrite cogs in a
memory-conscious way.  The programs here are meant to be glued together
with e.g. socat, ucspi-{tcp,unix,...} and operate on streams without
uncontrolled object instantiation.  (Possible exception: the
zrangebyscore call in analyze-logs.)

I've run these programs on the production ArchiveBot Redis updates
channel, and have noticed much more acceptable memory usage over time.
Next step: disable the LogAnalyzer actor, replace its duties, and see
just how we do.