mozilla-services/heka — pull request #743 by davidbirdsong (2014-03-19 18:29:21+00:00)
Title: general purpose file buffering for output plugins

I'd like to use the file buffering found in tcp_output.go in my own kakfa output module I'm currently assembling.

I'd aim to stay true to most of what tcp_output.go is doing, so my proposed changes are pretty simple:
```go
//only slightly modified TcpOutput struct, removes address and net objects, references TickerInterval directly
type BufferedHelper struct {
processMessageCount int64
sentMessageCount int64
readOffset int64
TickerInterval uint
parser *MessageProtoParser
writeFile *os.File
writeId uint
readFile *os.File
readId uint
checkpointFilename string
checkpointFile *os.File
queue string
name string
reportLock sync.Mutex
}

func (bh *BufferedHelper) Run(or OutputRunner, h PluginHelper, sender BufferedSender) (err error)
// copy/move code from TcpOutput.Run
```

The BufferedSender interface is simple
```go
type BufferedSender interface {
SendRecord(record []byte) (err error)
}
```
The user creates a new BufferedHelper in their plugin's ```Init``` function and then passes full control to BufferedHelper in their ```Run```.

I didn't see any code paths that allow TcpOutput.StreamOutput to exit on any failures pertaining to sending the record--only errors related to the file reading problems. If there's a need to add net related error exits, we could extend the interface to allow the user's plugin to signal an error exit.

```go
type BufferedSender interface {
SendRecord(record []byte) (err error)
Fatal() (err error)
}
```

I'm not attached to the names at all.