influxdb/influxdb — pull request #583 by chobie (2014-05-27 13:16:38+00:00)
Title: [RFC] Half duplex binary protocol plugin

Hi, I'm writing binary protocol as for I like fast insert api. I'd like to import this feature to core if it worth.

Feedback is appreciated.

### My Goal

* provide fast insert api.
* provide expandable, easy to implement protocol
* provide state full protocol. (don't need to send auth info every time)

Also, this binary protocol might useful for daily operation (like dump or batch import series).

This binary protocol is as is (not perfect). but it works fine for me.

#### Status

*Under Development*

TODO
[] Taking Feedback, adjust spec
[] Write test cases
[] Improve implementations,

#### Supported transfer protocol

* TCP v4
* Unix Domain Socket

This protocol uses socket in a half-duplex fashion. as for most client implementation getting very easy.

very rough client is here.
https://github.com/chobie/example-influxdb-binary-protocol-client

#### Protocol Basics

```
|4bytes length | protobuf payload |
```

#### Handshake

Binary protocol API requires handshake before sending any commands.

[Sequence]

* 1) Server send initial request to the client when Client connect to the server.
* 2) Client parse the message and send a response with authenticate information.
* 3) Server verify user. if it okay, Server returns message. otherwise sever will close connection.

```
| 1) Server | Server sends information.
| Greeting |
|
| 2) Client | Clients send credentials and database name.
| Greeting |
|
| 3) Server | Authenticate User as cluster admin or DB user
| Greeting |

these files has details.
https://github.com/chobie/example-influxdb-binary-protocol-client/blob/master/src/tcp/tcp.proto
https://github.com/chobie/example-influxdb-binary-protocol-client/blob/master/src/influxdb/client.go#L25

# After this sequence. client could send any commands.
```

#### Supported Commands

* writeSeries
* query
* createDatabase(s)
* dropDatabase(s)
* listDatabase
* ping (client may send ping command to keep connection.)
* close

Off course , we could implement more commands. for now, above api are available.

#### Configurations

```toml
[input_plugins]

# Configure the tcp binary protocol api
[input_plugins.tcp]
enabled = false
port = 8090
# provides unix domain listener
socket = "/tmp/influxd.sock"
timeout = 10s
write-buffer-size = 1024
read-buffer-size = 1024
chunked-response-limit = 1000
```

#### (reference) Simple writing benchmark

This benchmark is not fair but good start at beggining discussion for binary protocol.
Writing single points which has 3 fields 4096 times from single client.

```golang
// pseudo code
for i := 0; i < 4096; i++ {
err = client.WriteSeries(&protocol.Series{
Name: proto.String("debug"),
Fields: []string{"debug", "sever", "point"},
Points: []*protocol.Point{
&protocol.Point{
Values: []*protocol.FieldValue{
&protocol.FieldValue{StringValue: proto.String("helo")},
&protocol.FieldValue{StringValue: proto.String("main")},
&protocol.FieldValue{DoubleValue: proto.Float64(3.5)},
},
},
},
})
if err != nil {
fmt.Printf("Error: %+v\n", err)
return
}
}
```

```
# PHP
Json API (HTTP/1.1 keepalived): elapsed:2.593591 sec(0.000633 / per req)
Protobuf API(UnixDomainSocket): elapsed:1.590571 sec(0.000388 / per req)

# Go
go-influxdb: 0.30s user 0.22s system 41% cpu 1.274 total

go-tcp-experimental: 0.12s user 0.20s system 41% cpu 0.784 total
go-unix-experimental: 0.11s user 0.16s system 43% cpu 0.628 total
```

Approximately 1.5x faster than http api on my Mac (Early 2011 Core i7 2.2Ghz/ 16Gb Memory).

Note: Performance depends protobuf implementation.