Project: Kubernetes (Orchestration & Management / Scheduling & Orchestration)
File: PROTOCOL.md

The ttrpc protocol is client/server protocol to support multiple request streams over a single connection with lightweight framing. The client represents the process which initiated the underlying connection and the server is the process which accepted the connection. The protocol is currently defined as asymmetrical, with clients sending requests and servers sending responses. Both clients and servers are able to send stream data. The roles are also used in determining the stream identifiers, with client initiated streams using odd number identifiers and server initiated using even number. The protocol may be extended in the future to support server initiated streams, that is not supported in the latest version. The ttrpc protocol is designed to be lightweight and optimized for low latency and reliable connections between processes on the same host. The protocol does not include features for handling unreliable connections such as handshakes, resets, pings, or flow control. The protocol is designed to make low-overhead implementations as simple as possible. It is not intended as a suitable replacement for HTTP2/3 over the network. Each Message Frame consists of a 10-byte message header followed by message data. The data length and stream ID are both big-endian 4-byte unsigned integers. The message type is an unsigned 1-byte integer. The flags are also an unsigned 1-byte integer and use is defined by the message type. ++ | Data Length (32) | ++ | Stream ID (32) | ++--+ | Msg Type (8) | ++ | Flags (8) | ++--+ | Data (*) | ++ The Data Length field represents the number of bytes in the Data field. The total frame size will always be Data Length + 10 bytes. The maximum data length is 4MB and any larger size should be rejected. Due to the maximum data size being less than 16MB, the first frame byte should always be zero. This first byte should be considered reserved for future use. The Stream ID must be odd for client initiated streams and even for server initiated streams. Server initiated streams are not currently supported. | Message Type | Name | Description | |--|-|-| | 0x01 | Request | Initiates stream | | 0x02 | Response | Final stream data and terminates | | 0x03 | Data | Stream data | The request message is used to initiate stream and send along request data for properly routing and handling the stream. The stream may indicate unary without any inbound or outbound stream data with only a response is expected on the stream. The request may also indicate the stream is still open for more data and no response is expected until data is finished. If the remote indicates the stream is closed, the request may be considered non-unary but without anymore stream data

In the case of `remote closed`, the remote still expects to receive a response or stream data. For compatibility with non streaming clients, a request with empty flags indicates a unary request. | Flag | Name | Description | ||--|--| | 0x01 | `remote closed` | Non-unary, but no more data expected from remote | | 0x02 | `remote open` | Non-unary, remote is still sending data | The response message is used to end a stream with data, an empty response, or an error. A response message is the only expected message after a unary request. A non-unary request does not require a response message if the server is sending back stream data. A non-unary stream may return a single response message but no other stream data may follow. No response flags are defined at this time, flags should be empty. The data message is used to send data on an already initialized stream. Either client or server may send data. A data message is not allowed on a unary stream. A data message should not be sent after indicating `remote closed` to the peer. The last data message on a stream must set the `remote closed` flag. The `no data` flag is used to indicate that the data message does not include any data. This is normally used with the `remote closed` flag to indicate the stream is now closed without transmitting any data. Since ttrpc normally transmits a single object per message, a zero length data message may be interpreted as an empty object. For example, transmitting the number zero as a protobuf message ends up with a data length of zero, but the message is still considered data and should be processed. | Flag | Name | Description | ||--|--| | 0x01 | `remote closed` | No more data expected from remote | | 0x04 | `no data` | This message does not have data | All ttrpc requests use streams to transfer data. Unary streams will only have two messages sent per stream, a request from a client and a response from the server. Non-unary streams, however, may send any numbers of messages from the client and the server. This makes stream management more complicated than unary streams since both client and server need to track additional state. To keep this management as simple as possible, ttrpc minimizes the number of states and uses two flags instead of control frames. Each stream has two states while a stream is still alive: `local closed` and `remote closed`. Each peer considers local and remote from their own perspective and sets flags from the other peer's perspective. For example, if a client sends a data frame with the `remote closed` flag, that is indicating that the client is now `local closed` and the server will be `remote

A unary operation does not need to send these flags since each received message always indicates `remote closed`. Once a peer is both `local closed` and `remote closed`, the stream is considered finished and may be cleaned up. Due to the asymmetric nature of the current protocol, a client should always be in the `local closed` state before `remote closed` and a server should always be in the `remote closed` state before `local closed`. This happens because the client is always initiating requests and a client always expects a final response back from a server to indicate the initiated request has been fulfilled. This may mean server sends a final empty response to finish a stream even after it has already completed sending data before the client. +--+ +--+ | Client | | Server | ++-+ +-++ | ++ | local >+ Request +--> remote closed | ++ | closed | | | +-+ | finished <--+ Response +--< finished | +-+ | | | RC: `remote closed` flag RO: `remote open` flag +--+ +--+ | Client | | Server | ++-+ +-++ | +--+ | >-+ Request [RO] +--> | +--+ | | | | ++ | >--+ Data +> | ++ | | | | +--+ | local >+ Data [RC] +> remote closed | +--+ | closed | | | +-+ | finished <--+ Response +--< finished | +-+ | | | +--+ +--+ | Client | | Server | ++-+ +-++ | +--+ | local >-+ Request [RC] +--> remote closed | +--+ | closed | | | ++ | <--+ Data +< | ++ | | | | +--+ | finished <+ Data [RC] +< finished | +--+ | | | +--+ +--+ | Client | | Server | ++-+ +-++ | +--+ | >-+ Request [RO] +--> | +--+ | | | | ++ | >--+ Data +> | ++ | | | | ++ | <--+ Data +< | ++ | | | | ++ | >--+ Data +> | ++ | | | | +--+ | local >+ Data [RC] +> remote closed | +--+ | closed | | | ++ | <--+ Data +< | ++ | | | | +--+ | finished <+ Data [RC] +< finished | +--+ | | | While this protocol is defined primarily to support Remote Procedure Calls, the protocol does not define the request and response types beyond the messages defined in the protocol. The implementation provides a default protobuf definition of request and response which may be used for cross language rpc. All implementations should at least define a request type which support routing by procedure name and a response type which supports call status. | Version | Features | ||| | 1.0 | Unary requests only | | 1.2 | Streaming support |