snowplow/snowplow — pull request #580 by pkallos (2014-03-24 00:27:31+00:00)
Title: Use Future for writing to sink.

Performance improvement to `scala-stream-collector`.

I applied this patch `StdoutSink` to simulate a 50ms delay in writing to the sink (the Kinesis sink uses Await to wait for the response and log it).
```diff
@@ -36,10 +36,19 @@ import com.typesafe.config.Config
import scalastream._
import thrift.SnowplowRawEvent

+import scala.concurrent._
+import scala.concurrent.duration._
+import ExecutionContext.Implicits.global
+
class StdoutSink extends AbstractSink {
// Print a Base64-encoded event.
def storeRawEvent(event: SnowplowRawEvent, key: String) = {
- println(Base64.encodeBase64String(serializeEvent(event)))
+ val decode = Future {
+ Thread.sleep(50)
+ Base64.encodeBase64String(serializeEvent(event))
+ }
+ val result = Await.result(decode, Duration(60, SECONDS))
+ println(result)
null
}
}
```

Here is the performance running siege before my change:
```
siege -b -c30 -t60s http://localhost:8080/i
Transactions:	1135 hits
Availability:	74.43 %
Elapsed time:	59.30 secs
Data transferred:	0.04 MB
Response time:	1.20 secs
Transaction rate:	19.14 trans/sec
Throughput:	0.00 MB/sec
Concurrency:	23.04
Successful transactions: 1135
Failed transactions:	390
Longest transaction:	2.52
Shortest transaction:	0.06
```
Seems like it was halting quite badly, so pulling the sink write out of the scope of the HTTP request means it's able to respond much faster.
```
Transactions:	32591 hits
Availability:	100.00 %
Elapsed time:	59.40 secs
Data transferred:	1.15 MB
Response time:	0.05 secs
Transaction rate:	548.67 trans/sec
Throughput:	0.02 MB/sec
Concurrency:	25.96
Successful transactions: 32591
Failed transactions:	0
Longest transaction:	13.57
Shortest transaction:	0.00
```