add --files-from=FILE to allow transfer of select files only

This solves the change of behavior introduced by processing
excludes/includes during os.walk(), where previously:

s3cmd sync --exclude='*' --include='*.gpg'

would walk the whole tree and transfer only the files named *.gpg.

Since the change to os.walk(), the exclude '*' matches everything, and
nothing is transferred.

This patch introduces --files-from=FILE to match rsync behaviour,
where the list of files to transfer (local to remote) is taken not
from an os.walk(), but from the explicit list in FILE.

The equivalent for remote to local, and remote to remote, is not yet
implemented.
accept --files-from=- to read from stdin

This allows shell syntax:

find . -name \*.gpg | s3cmd sync --files-from=- src dst

to take the list of files to transfer from stdin.

Be careful, as using with a --delete option will cause files on the
remote side not listed in stdin to be deleted too.
add --files-from to manpage
use --files-from only on local source (not remote, not dest)

The restriction to use --files-from on the source and not the
destination list comes from the equivalent behavior in rsync.

The restriction on not using it for remote sources is only because I
haven't figured out the best way to handle that.  That may be added in
the future.
Add option to ignore error on copy when key no longer exists
fixing the order of public URL components
Fix crash when cloudfront operations receive 500 errors.
Retries parameter was not being sent to send_request as a keyword arg.
Apply include/exclude to destinations too, via mdomsch/s3cmd@268e7c3bd37cf60920025d556449fc1b93a1c590 Refs s3tools/s3cmd#109.
Windows fix re / vs os.path.sep

Even on Windows where os.path.sep is backslash, forward-slash is the S3 separator character.
add --max-delete for del and sync commands, fixes bug #155

Fixes https://github.com/s3tools/s3cmd/issues/155

Similar to rsync --max-delete, allow the user to specify a maximum
number of objects to delete for the [del] and [sync] commands.  If
that number would be exceeded, output a warning message and do not
delete anything.  This prevents a simple invocation error
(e.g. launching from the wrong directory) from erasing all the data in
a bucket accidentally.

By default, this is disabled.  A user must explicitly specify
--max-delete=NUM for this protection to be available, just as in
rsync.
sync: ignore --delete-removed if no source files, bug #155

https://github.com/s3tools/s3cmd/issues/155

If --delete-removed is specified, yet no source files were found, all
files at destination are deleted.  This is more likely a user error
than the desired outcome.

This patch adds a check to [sync] for both --delete-removed and no
source files, and refuses to delete on the destination in this case.
The user can specify --force to force deletion if that is really what
is intended.
content_encoding != charset, bug #153

https://github.com/s3tools/s3cmd/issues/153

Thus spake Castedo Ellerman:
I believe the commit 64eb484 you did on 2012-07-31 04:04:24Z added
incorrect code that sets the HTTP Content-Encoding header to be a
character encoding (typically "UTF-8").

I just got bitten by this because I used a recent s3cmd to upload a
file and now a Perl script that uses the "get" function from
LWP::Simple to download that file fails because that file uploaded by
s3cmd got the meta-tagged to have "Content-Encoding" as "UTF-8" which
is an invalid value. This is the same root cause for this bug
affecting Richard Harrington:
  https://sourceforge.net/mailarchive/message.php?msg_id=30653510

The problem is that the code you added is confounding the "charset"
with "content encoding". Here is the definition of HTTP "content
encoding":
  http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.5
  http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.11
which is different and independent from charset (e.g. UTF-8).

The data member "self.config.encoding" of class S3 in S3.py is a
charset (looks like by default set to "utf-8").

In contrast the variable "content_encoding" originally added by
Karsten Sperling <mail@ksperling.net> is supposed to be an HTTP
content encoding.

It is with your merge that you incorrectly assign a charset
(self.config.encoding.upper()) to the variable content_encoding which
is intended to be used at an HTTP Contend-Encoding header and use
content_encoding as a charset for the
content_type. self.config.encoding.upper() and content_encoding should
remain completely independent of each other, they are two different
things.

This patch from Castedo fixes the mistake.
provide a better exception / error reporting message, bug #158

Trying to cut down on the duplicates to the -bugs mailing list.
Reverted "/usr/bin/python" -> "/usr/bin/env python"

Partially reverted af984ba9
Merge branch 'master' of ssh://github.com/s3tools/s3cmd
Merge pull request #159 from mdomsch/bug/158

provide a better exception / error reporting message, bug #158
Merge pull request #154 from EnlytonJ/master

Fix needed on Windows where os.path.sep != s3_separator
Relaxed timeout for http connection during IAM role lookup
Fix printing error in `report_exception(e)`

Fixes `TypeError: unsupported operand type(s) for %: 'NoneType' and 'str'` arising from using string formatting with the /result/ of sys.stderr.write() instead of the argument to it.
Merge branch 'files-from' into merge, pull request #116