elasticsearch/logstash — pull request #1755 by DanielRedOak (2014-09-19 01:57:13+00:00)
Title: S3 input: add exclude pattern, backup_add_prefix, cleanup backup methods, and use aws_config from plugin mixins

This commit cleans up how backups to S3 buckets happen and adds a few neat features.

First off, when backing up to an S3 bucket, instead of downloading the file locally then uploading it back to the backup bucket, this input will now use the copy_to and move_to instance methods directly saving bandwidth and potentially cost.

The input also now uses the standard aws_config from the mixin plugins. The :credentials config option is now deprecated.

Feature wise, I added the ability to exclude keys matching a ruby regex, added the ability to add a prefix to keys when backing them up (especially useful if you want to backup to the same bucket you source from), and made sure that if backing up to the same bucket and using a backup_add_prefix that we automatically exclude keys starting with the backup_add_prefix. This prevents duplicates and loops of terribleness.