graphite ingest write data in batches to coordinator

should, in theory, be a lot more efficient. however this is very hard to gauge right now, since we have no instrumentation.
I did however run a bunchmark. it basically writes 1000 datapoints per "request" (i.e. 1000 lines in the graphite protocol for unique metric keys, with random values and timestamps), with 10 parallell connections, and for each it does this 100 times. and it measures how quick each connection completes.  I also looked in the corresponding logs.
every benchmark run is done with cleared wal and db directories, and a fresh log file.
i ran both the before and after, each twice, with similar results. so only one result is shown.


before (current implementation)
```
[dieter.plaetinck@dfvimeographite1 tomselleck]$ head -n 1000  generate_proto1_metrics.txt | ./tomselleck -addr :2003 -workers 10 -requests 100 -maxprocs 2
2014/04/15 19:02:28 {<nil>:2003 10 100 false false} 1000

Tom Selleck Reports:

Address to hit: :2003 (<nil>:2003)
 * 10 workers
 * 100 reqs per worker

Connects: 1000
ConnectsErrors: 0
BytesWritten: 43909000
BytesRead: 0
BytesReadErrors: 0
EmptyResponses: 1000
UnexpectedLengthResponses: 0

Breakdown by percentile:
Pctl       Best        Wrst
50%    58.362us   430.623us
60%   431.236us   498.763us
70%   498.821us   574.047us
80%   575.278us   667.932us
90%   669.011us    739.66us
95%   739.977us   755.701us
96%    757.85us   781.566us
97%   789.524us   914.632us
98%   915.604us  3.427624ms
99%  3.468484ms  4.615237ms
All    58.362us  4.615237ms

Total time: 43.932527ms
Reqs per sec: 22762.178
Avg req time: 397.22us
Median req time: 355.848us

```

the log file ends with
```
(...)
[04/15/14 19:11:19] [INFO] Opening log file /opt/influxdb/shared/data/wal/log.10001
[2014/04/15 19:11:19 EDT] [EROR] (api/graphite.(*Server).Serve:78) GraphiteServer: Accept: %!(EXTRA *net.OpError=accept tcp [::]:2003: too many 
[04/15/14 19:11:19] [EROR] GraphiteServer: Accept: %!(EXTRA *net.OpError=accept tcp [::]:2003: too many open files)
[2014/04/15 19:11:19 EDT] [EROR] (api/graphite.(*Server).Serve:78) GraphiteServer: Accept: %!(EXTRA *net.OpError=accept tcp [::]:2003: too many 
[04/15/14 19:11:19] [EROR] GraphiteServer: Accept: %!(EXTRA *net.OpError=accept tcp [::]:2003: too many open files)
[2014/04/15 19:11:19 EDT] [EROR] (api/graphite.(*Server).Serve:78) GraphiteServer: Accept: %!(EXTRA *net.OpError=accept tcp [::]:2003: too many 
[04/15/14 19:11:19] [EROR] GraphiteServer: Accept: %!(EXTRA *net.OpError=accept tcp [::]:2003: too many open files)
[2014/04/15 19:11:19 EDT] [EROR] (api/graphite.(*Server).Serve:78) GraphiteServer: Accept: %!(EXTRA *net.OpError=accept tcp [::]:2003: too many 
[2014/04/15 19:11:19 EDT] [EROR] (api/graphite.(*Server).Serve:78) GraphiteServer: Accept: %!(EXTRA *net.OpError=accept tcp [::]:2003: too many 
[2014/04/15 19:11:19 EDT] [EROR] (api/graphite.(*Server).Serve:78) GraphiteServer: Accept: %!(EXTRA *net.OpError=accept tcp [::]:2003: too many 
[04/15/14 19:11:19] [EROR] Cannot write bookmark open /opt/influxdb/shared/data/wal/bookmark.new: too many open files
panic: open /opt/influxdb/shared/data/wal/log.10001: too many open files

goroutine 145 [running]:
runtime.panic(0x85df00, 0xc210bffe40)
        /home/vagrant/bin/go/src/pkg/runtime/panic.c:266 +0xb6
wal.(*WAL).AssignSequenceNumbersAndLog(0xc210071d80, 0xc2119e9c00, 0x7f293c933ff8, 0xc2101121c0, 0x7f293c91de00, ...)
        /home/vagrant/influxdb/src/wal/wal.go:394 +0x10a
cluster.(*ShardData).Write(0xc2101121c0, 0xc2119e9c00, 0xc211428b70, 0x7f29282ffae8)
        /home/vagrant/influxdb/src/cluster/shard.go:183 +0x96
coordinator.(*CoordinatorImpl).write(0xc2100a4860, 0xc210099390, 0x8, 0xc211a9cb70, 0x1, ...)
        /home/vagrant/influxdb/src/coordinator/coordinator.go:634 +0xb2
coordinator.(*CoordinatorImpl).CommitSeriesData(0xc2100a4860, 0xc210099390, 0x8, 0xc211a9cb58, 0x1, ...)
        /home/vagrant/influxdb/src/coordinator/coordinator.go:622 +0x8a1
coordinator.(*CoordinatorImpl).WriteSeriesData(0xc2100a4860, 0x7f293c933f18, 0xc2100bc440, 0xc210099390, 0x8, ...)
        /home/vagrant/influxdb/src/coordinator/coordinator.go:462 +0x1d3
api/graphite.(*Server).writePoints(0xc2100467e0, 0xc2116e2140, 0x0, 0x0)
        /home/vagrant/influxdb/src/api/graphite/api.go:100 +0xf5
api/graphite.(*Server).handleClient(0xc2100467e0, 0x7f293c933e90, 0xc210135b28)
        /home/vagrant/influxdb/src/api/graphite/api.go:145 +0x482
created by api/graphite.(*Server).Serve
        /home/vagrant/influxdb/src/api/graphite/api.go:81 +0x175
```

with this code applied, the benchmark shows:
(note that the higher percentiles perform much worse. the total median is in the same order, but the bad performing ones drive up the average)
```
[dieter.plaetinck@dfvimeographite1 tomselleck]$ head -n 1000  generate_proto1_metrics.txt | ./tomselleck -addr :2003 -workers 10 -requests 100 -maxprocs 2
2014/04/15 18:57:18 {<nil>:2003 10 100 false false} 1000

Tom Selleck Reports:

Address to hit: :2003 (<nil>:2003)
 * 10 workers
 * 100 reqs per worker

Connects: 1000
ConnectsErrors: 0
BytesWritten: 43909000
BytesRead: 0
BytesReadErrors: 0
EmptyResponses: 1000
UnexpectedLengthResponses: 0

Breakdown by percentile:
Pctl       Best        Wrst
50%    55.122us   438.014us
60%   438.227us   498.452us
70%   499.385us   559.652us
80%   559.988us   668.483us
90%   670.838us  1.175131ms
95%2.999352913s2.999781967s
96%2.999783397s2.999955982s
97%2.999960642s3.000136881s
98%3.000152211s3.000300858s
99%3.000318443s3.000749172s
All    55.122us3.000749172s

Total time: 15.03814515s
Reqs per sec: 66.498
Avg req time: 150.364459ms
Median req time: 405.253us

```

and the log shows EOF's (but no panic like before ;-)
```
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[04/15/14 19:15:56] [EROR] EOF
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[04/15/14 19:15:56] [EROR] EOF
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[04/15/14 19:15:56] [EROR] EOF
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[04/15/14 19:15:56] [INFO] Opening log file /opt/influxdb/shared/data/wal/log.1
[2014/04/15 19:15:56 EDT] [INFO] (wal.(*WAL).openLog:360) Opening log file /opt/influxdb/shared/data/wal/log.1
[04/15/14 19:15:56] [INFO] Opening index file /opt/influxdb/shared/data/wal/index.1
[2014/04/15 19:15:56 EDT] [INFO] (wal.(*WAL).openLog:374) Opening index file /opt/influxdb/shared/data/wal/index.1
[04/15/14 19:15:56] [EROR] EOF
[04/15/14 19:15:56] [EROR] EOF
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[04/15/14 19:15:56] [EROR] EOF
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[04/15/14 19:15:56] [EROR] EOF
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[04/15/14 19:15:56] [EROR] EOF
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[04/15/14 19:15:56] [EROR] EOF
[2014/04/15 19:15:56 EDT] [EROR] (api/graphite.(*Server).handleClient:175) EOF
[04/15/14 19:15:56] [EROR] EOF
```


i haven't really analyzed this much yet.  go makes it easy to add cpu and memory profiling  (see for example https://github.com/vimeo/statsdaemon/blob/master/statsdaemon.go#L553-568) so I could add that, and see what's going on. once we have a facility in the code base to track metrics i could instrument the graphite stuff and `coordinator.WriteSeriesData`