snowplow/snowplow — pull request #546 by pkallos (2014-03-03 20:46:38+00:00)
Title: Initial implementation of kinesis-s3-sink.

Moving this PR over to snowplow/master

Here's my preliminary implementation of a kinesis -> s3 raw event sink, re #506.

This is my first "from scratch" scala application so feel encouraged to rip it apart with constructive criticism and suggestions.

What I did:
* Forked upworthy/amazon-kinesis-connectors to grab their pomfile, then modified it to work with the github maven plugin. For now I'm hosting the amazon-kinesis-connectors mvn repo on github. **What's the best place to host this?** -- this was my first experience packaging something in maven, not nearly as easy as rubygems or nodejs libs ;)
* Used the Java `amazon-kinesis-connectors` library to build an app that pulls from a kinesis stream and flushes out to s3 every so often.
* I copied the naming scheme in the [example kinesis connector](https://github.com/pkallos/amazon-kinesis-connectors/blob/master/src/main/java/com/amazonaws/services/kinesis/connectors/s3/S3Emitter.java) for files that are stored in S3. Would be nice to go to a `year=YYYY/month=MM/day=DD/hour=HH` scheme and have ETL jobs that support this.

Things I'm not sure about:
* My `built.sbt` -- **does this conform to the project guidelines ? Is this a sane way of doing it?**
* For some reason that I can't decipher there's an issue when I run sbt assembly on my box, something about the scala version. When I run `java -jar target/scala.2.9.1/kinesis-s3-sink` it whines `Error: Could not find or load main class com.snowplowanalytics.snowplow.sinks.kinesiss3.SinkApp` ... Something I could fix with some googling but any hints? Have been using `sbt run` in the meantime.
* I didn't really write a Scala wrapper for `amazon-kinesis-connectors` library as was suggested in the ticket, mostly because I wasn't sure what the "prettiest" way to do it was. Instead I just used the library as is, hope that's okay otherwise I could use some guidance as to how to write the Scala wrapper.
* Because I'm new to Scala I'm pretty sure my code looks very Javaish. **Any suggestions on how to make this code more Scala-idiomatic would be appreciated.**
* Class structure + naming conventions + project directory structure

TODOs I can think of:
* Add a README.md
* Add support for specifying access keys in the config file -- right now it's hardcoded to use `DefaultAWSCredentialsProviderChain` which uses the first of: *Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_KEY, Java System Properties - aws.accessKeyId and aws.secretKey, Instance profile credentials delivered through the Amazon EC2 metadata service*
* Add support for some of the other relevant [configuration options](https://github.com/pkallos/amazon-kinesis-connectors/blob/master/src/main/java/com/amazonaws/services/kinesis/connectors/KinesisConnectorConfiguration.java).
* ~~Add comments and method doc~~
* Probably refactor this code based on feedback

The AWS permissions required to run this app are:
* CloudWatch -- I tested with full access
* DynamoDB -- I tested with full access. Needs to be able to create table, write to that table. The Kinesis library does checkpointing by creating a Table called "application-name" and periodically updating a key that stores the checkpoint.
* Kinesis -- I tested with read/write access to a specific stream. Needs at least permission to read from the stream specified in the config.
* S3 -- I tested with read/write access to a specific S3 bucket. At least permission to write to the configured S3 bucket.

Also I plan on rebasing this branch over-and-over as I clean up so if you fork from it be prepared to diverge :)