Clever/baseworker-go — pull request #13 by kvigen (2014-08-29 20:02:48+00:00)
Title: pathio package for handling different data paths

pahtio supports writing to and reading from different types of paths
transparently. It supports three types currently:
1. Empty (stdin/stdout)
2. Local file paths
3. S3 File Paths

It has both a read and a write interface. I've tested the interface for both of the workers I need it for (oplog-dump / oplog-replay) and it works nicely for both of them.

Unfortunately, since s3 needs
to know the length of a file before starting to uploading it it's a bit
awkward to expose a writer interface directly because that would involve
buffering data in memory. Instead we provide a writer interface that
takes in a reader which seems both reasonable and can be transformed by
the caller using io.Pipe()