aws/aws-cli — pull request #673 by jamesls (2014-02-24 19:21:09+00:00)
Title: Fix hanging Ctrl+C on S3 downloads

This is a regression that was introduced when the IO queue/thread
was introduced (https://github.com/aws/aws-cli/issues/619).

The current shutdown order of the threads is:

1. Queue end sentinel for IO thread.
2. Queue end sentinel for result thread.
3. Queue end sentinel for worker threads.
4. .join() threads in this order:
[io_thread, result_thread [, worker threads]]

Though the actual thread shutdown order is non-deterministic,
it's fairly common that the threads shutdown in roughly the above
order. This means that the IO thread will generally shutdown before
all the worker threads have shutdown.

However, the download tasks can still be enqueueing writes to the
IO queue. If the IO thread shutsdown there's nothing consuming
writes on the other end of the queue. Given that the queue is
bounded in maxsize, .put() calls to the queue will block until space
becomes available. This will never happen if the IO queue is already
shutdown.

The fix here is to ensure that the IO thread is always the last thing
to shutdown. This means any remaining IO requests will be executed
before the IO thread shutsdown. This will prevent deadlock.

Added unit tests demonstrates this issue. I've also added an
integration test that actually sends a SIGINT to the process
and verifies it exits in a timely manner so can ensure we
don't regress on this again.

Note: some unit/integ tests needed updating because they were
using .call() multiple times.

Fixes #650
Fixes #657