Draft: draft-thomson-http-binary-message

Commentary: This is fully - featured , which is a little awkward . We can probably trim this down a little once we have a better handle on the requirements . 
 is the idea here that padding might just be a separate field for each ( un)known - length message ? 
 Not sure if you meant to ask this on instead . The idea I had for padding is that the format as described is self - delimiting . ( Actually , it was accidental , and I only realized that it worked that way after coming back and thinking about padding . ) Therefore , you can just add padding at the end . We can mandate the use of zero bytes , which works for the truncated message syntax I suggest here . 
 The format I 've sketched out for binary messages can be padded with zeros without needing a separate padding field . That 's probably a feature worth mentioning in both drafts . 
 Padding is in the binary HTTP draft now .
This document defines a simple format for representing an HTTP
message (HTTP), either request or response.  This allows for the
encoding of HTTP messages that can be conveyed outside of an HTTP
protocol.  This enables the transformation of entire messages,
including the application of authenticated encryption.
This format is informed by the framing structure of HTTP/2 (H2) and
HTTP/3 (H3).  In comparison, this format simpler by virtue of not
including either header compression (HPACK, QPACK) or a generic
framing layer.
This format provides an alternative to the "message/http" content
type defined in MESSAGING.  A binary format permits more efficient
encoding and processing of messages.  A binary format also reduces
exposure to security problems related to processing of HTTP messages.
an indefinite-length encoding enables efficient generation of
messages where lengths are not known when encoding starts.
This document uses terminology from HTTP (HTTP) and notation from
QUIC (QUIC).
Control data.  For a request, this contains the request method and
target.  For a response, this contains the status code.
Header section.  This contains zero or more header fields.
Content.  This is a sequence of zero or more bytes.
Trailer section.  This contains zero or more trailer fields.
A response message can include multiple informational responses,
which causes the control data and header section to be repeated.
All lengths and numeric values are encoded using the variable-length
integer encoding from QUIC.
A message that has a known length at the time of construction uses
the format shown in format-known-length.
That is, a known-length message consists of a framing indicator, a
block of control data that is formatted according to the value of the
framing indicator, a header section with a length prefix, binary
content with a length prefix, and a trailer section with a length
prefix.
Fields in the header and trailer sections consist of a length-
prefixed name and length-prefixed value.  Both name and value are
sequences of bytes that cannot be zero length.
The format allows for the message to be truncated before any of the
length prefixes that precede the field sections or content.  This
reduces the overall message size.  A message that is truncated at any
other point is invalid; see invalid.
The variable-length integer encoding means that there is a limit of
2^62-1 bytes for each field section and the message content.
That is, an indeterminate length consists of a framing indicator, a
block of control data that is formatted according to the value of the
framing indicator, a header section that is terminated by a zero
value, any number of non-zero-length chunks of binary content, a zero
value, and a trailer section that is terminated by a zero value.
Indeterminate-length messages can be truncated in a similar way as
known-length messages.  Truncation occurs after the control data, or
after the Content Terminator field that ends a field section or
sequence of content chunks.  A message that is truncated at any other
point is invalid; see invalid.
Indeterminate-length messages use the same encoding for field lines
as known-length messages; see fields.
A value of 0 describes a request of known length.
A value of 1 describes a response of known length.
A value of 2 describes a request of indeterminate length.
A value of 3 describes a response of indeterminate length.
Other values cause the message to be invalid; see invalid.
The control data for a request message includes four values that
correspond to the values of the ":method", ":scheme", ":authority",
and ":path" pseudo-header fields described in HTTP/2 (Section 8.1.2.3
of H2).  These fields are encoded, each with a length prefix, in the
order listed.
The rules in Section 8.1.2.3 of H2 for constructing pseudo-header
fields apply to the construction of these values.  However, where the
":authority" pseudo-header field might be omitted in HTTP/2, a zero-
length value is encoded instead.
The format of request control data is shown in format-request-
control-data.
The control data for a request message includes a single field that
corresponds to the ":status" pseudo-header field in HTTP/2 H2.  This
field is encoded as a single variable length integer, not a decimal
string.
The format of final response control data is shown in format-
response-control-data.
This format supports informational status codes (see Section 15.2 of
HTTP).  Responses that include information status codes are encoded
by repeating the response control data and associated header section
until the final status code is encoded.
The format of the informational response control data is shown in
format-informational.
A response message can include any number of informational responses.
If the response control data includes an informational status code
(that is, a value between 100 and 199 inclusive), the control data is
followed by a header section (encoded with known- or indeterminate-
length according to the framing indicator).  After the header
section, another response control data block follows.
Header and trailer sections consist of zero or more field lines; see
Section 5 of HTTP.  The format of a field section depends on whether
the message is known- or intermediate-length.
Each field line includes a name and a value.  Both the name and value
are non-zero length sequences of bytes.  The format of a field line
is shown in format-field-line.
For field names, byte values that are not permitted in an HTTP field
name cause the message to be invalid; see Section 5.1 of HTTP for a
definition of what is valid and invalid for handling of invalid
messages.
In addition, values from the ASCII uppercase range (0x41-0x5a
inclusive) MUST be translated to lowercase values (0x61-0x7a) when
generating or forwarding messages.  A recipient MUST treat a message
containing field names with bytes in the range 0x41-0x5a as invalid;
see invalid.
For field values, byte values that are not permitted in an HTTP field
value cause the message to be invalid; see Section 5.5 of HTTP for a
definition of valid values.
The same field name can be repeated in multiple field lines; see
Section 5.2 of HTTP for the semantics of repeated field names and
rules for combining values.
Like HTTP/2, this format has an exception for the combination of
multiple instances of the "Cookie" field.  Instances of fields with
the ASCII-encoded value of "cookie" are combined using a semicolon
octet (0x3b) rather than a comma; see Section 8.1.2.5 of H2.
This format provides fixed locations for content that would be
carried in HTTP/2 pseudo-fields.  Therefore, there is no need to
include field lines containing a name of ":method", ":scheme",
":authority", ":path", or ":status".  Fields that contain one of
these names cause the message to be invalid; see invalid.  Pseudo-
fields that are defined by protocol extensions MAY be included,
however field lines containing pseudo-fields MUST precede other field
lines.
The content of messages is a sequence of bytes of any length.  Though
a known-length message has a limit, this limit is large enough that
it is unlikely to be a practical limitation.  There is there is no
limit to an indeterminate length message.
Omitting content by truncating a message is only possible if the
content is zero-length.
This document describes a number of ways that a message can be
invalid.  Invalid messages MUST NOT be processed except to log an
error and produce an error response.
The format is designed to allow incremental processing.
Implementations need to be aware of the possibility that an error
might be detected after performing incremental processing.
This section includes example requests and responses encoded in both
known-length and indefinite-length forms.
The example HTTP/1.1 message in ex-request shows the content of a
"message/http".
Valid HTTP/1.1 messages require lines terminated with CRLF (the two
bytes 0x0a and 0x0d).  For simplicity and consistenct, the content of
these examples is limited to text, which also uses CRLF for line
endings.
This can be expressed as a binary message (type "message/bhttp")
using a known-length encoding as shown in hexadecimal in ex-bink-
request.  ex-bink-request view includes some of the text alongside to
show that most of the content is not modified.
This example shows that the Host header field is not replicated in
the :authority field, as is required for ensuring that the request is
reproduced accurately; see Section 8.1.2.3 of H2.
The same message can be truncated with no effect on interpretation.
In this case, the last two bytes - corresponding to content and a
trailer section - can each be removed without altering the semantics
of the message.
The same message, encoded using an indefinite-length encoding is
shown in ex-bini-request.  As the content of this message is empty,
the difference in formats is negligible.
This indefinite-length encoding can be truncated by two bytes in the
same way.
Response messages can contain interim (1xx) status codes as the
message in ex-response shows. ex-response includes examples of
informational status codes defined in RFC2518 and RFC8297.
As this is a longer example, only the indefinite-length encoding is
shown in ex-bini-response.  Note here that the specific text used in
the reason phrase is not retained by this encoding.
A response that uses the chunked encoding (Section 7.1 of MESSAGING)
as shown for ex-chunked can be encoded by preserving chunk boundaries
using indefinite-length encoding, which minimizes buffering needed to
translate into the binary format.  However, these boundaries do not
need to be retained and any chunk extensions cannot be conveyed using
the binary format.
ex-bink-chunked shows this message using the known-length coding.
Note that the transfer-encoding header field is removed.
The message/http media type can be used to enclose a single HTTP
request or response message, provided that it obeys the MIME
restrictions for all "message" types regarding line length and
encodings.
Many of the considerations that apply to HTTP message handling apply
to this format; see Section 17 of HTTP and Section 11 of MESSAGING
for common issues in handling HTTP messages.
Strict parsing of the format with no tolerance for errors can help
avoid a number of attacks.  However, implementations still need to be
aware of the possibility of resource exhaustion attacks that might
arise from receiving large messages, particularly those with large
numbers of fields.
The format is designed to allow for minimal state when translating
for use with HTTP proper.  However, producing a combined value for
fields, which might be necessary for the "Cookie" field when
translating this format (like HTTP/1.1 MESSAGING), can require the
commitment of resources.  Implementations need to ensure that they
aren't subject to resource exhaustion attack from a maliciously
crafted message.
Please add the "Media Types" registry at
https://www.iana.org/assignments/media-types [1] with the
registration information in media-type for the media type "message/
bhttp".