Code change:
This modifies parsing to consider empty or not-present header fields as empty containers; serialisation omits them from output.
The SH syntax does not directly allow empty field values, such as defined for Accept, although this might be allowed by field definitions like MyField = [ sh-list ] Likewise, the list types do not allow for empty lists. sh-list = list-member *( OWS "," OWS list-member ) list-member = sh-item I think this is a very odd choice for a generic syntax because the empty set is often given semantic meaning in generic communication. It's hard to do math without zero. If this is intended, the rationale should be explained in the spec, and there should be some discussion with examples about defining fields as an optional value.
FWIW, even if empty lists may be problematic top-level, the situation is different when they are nested...
ietf104 - interest in the room in supporting empty (perhaps null?) syntax
I agree this is something important. It may be used to mean "I know I must pass the info but I don't know the info", which is different from "I don't care about the protocol" or passing a dummy value just to fill the hole.
In the original base material of header definitions, empty values were generally just not sent at all. Having a specific Null value may or not make sense, not sure.
I personally like the distinction between "known to be empty" and "not presented either because I don't know or because it's empty". Some semantics might explicitly require the presence of some fields which we would enforce by the protocol. This doesn't mean they can't be empty, they must be present, for example, to make sure they weren't dropped in the middle. For example, "Connection: foo" gets rid of the field, not just the value. Just my two cents, Willy
Do we have any RFC'ed headers where there is a difference between a non-present and an empty header ?
There's an accept-something if I remember well. Also I've been using "Host:" in many occasions with HTTP/1.1 requests when I didn't know the host name and this tends to work where no Host field fails ;-) Willy
I'm not against a '?N' null token (along the lines of true/false), but I have a hard time thinking of a use for it.
URL
Wouldn't an empty list be equivalent to an empty value, thus we don't need a new syntax?
Putting an empty list where for instance an integer is expected, just to say that there will be none, sounds counter-intuitive to me.
On Fri, Apr 05, 2019 at 01:37:11AM -0700, Julian Reschke wrote: Indeed, probably.
On Fri, Apr 05, 2019 at 03:11:00AM -0700, Poul-Henning Kamp wrote: Agreed in general. My point was that sometimes specs use "MUST send" when the sender doesn't know, resulting in stupid values appearing. That's exactly what led a number of agents to place a content-length with stupid values (0 or huge ones) with the CONNECT method a long time ago. An empty value where an integer is needed by the application is more of an application-level error than a protocol-level one in my opinion : it's up to the application to decide how to recover from this missing piece of information (is it really needed to process the request or not). For me it can mean "don't care", "don't know" or "not representable". After all floats support "NaN" for the same reason :-) Willy
If there is a "MUST send" requirement, there must also be something available in the same spec that can be sent ?
There is not, and never has been, an interop problem with sending empty field values. They are a list value of zero items or a string item of zero length. Accept and Host, respectively.
NAME how do you know that?
Because if a recipient drops an empty Host field the request will be rejected with 400. There have been no reports of such breakage in 25 years. It isn’t even a likely scenario given how message fields are parsed and reproduced.
If a recipient doesn't drop an empty host header, the request will still be rejected with 400, surely? Regardless, I'd note that many headers address this situation with a reserved value like . Is there value in standardising on one such value for future headers? I'm not convinced, but I'd note that we also have many headers that use to denote "all"; it seems to me that if we're going to do null, we should do that too (but again, I'm not convinced this is adding value).
On Mon, Apr 08, 2019 at 05:16:41PM -0700, Mark Nottingham wrote: It depends on the header field that was empty ;-) I've used the empty Host header field quite a bit (even in health check requests). My observation has been that while HTTP/1.1 without Host naturally causes a 400, HTTP/1.1 with empty Host is often processed similar to HTTP/1.0 without Host or as 1.1 with an unknown Host. I'd say "why not", but it's useful to consider lists and the iterative processing that can happen, removing one item at a time. Having to place a specific value to denote emptiness would require specific processing (i.e. check for emptiness after removing any item just for the sake of replacing it with the special value). By the way, speaking about lists makes me think we may complicate interop when a header field appears multiple times. For example if we have this in a request : List: a,b,c,d Foo: bar List: e,f,g,h It's not unreasonable to think that some simplified implementations may replace the first occurrence of List with "none" after processing it, leading to this : List: none Foo: bar List: e,f,g,h Or possibly this : List: none Foo: bar List: none Which may be processed as "List: none,e,f,g,h" or "List: none,none" respectively. Just my two cents, Willy
Willy, I think that would be clearly broken, and we should be able to catch it with tests. I think we're talking about all of the container types here (dict, list, list of lists, parameterised lists), correct? Julian, you gave as an example. Given that the behaviour recommended is exactly the same as if the header was missing, that's not very compelling. Besides the potential stripping issue (I have heard of implementations that strip empty headers, but don't have details on hand), some header libraries may not make a distinction between missing and empty headers. Overall, I don't see a lot of value in accommodating empty headers directly, and I don't think we should be promoting it as a practice.
Agree, I'm not convinced either. I see the potential, but Gettys rule applies: "The only thing worse than generalizing from one example is generalizing from no examples at all." (May actually be from Phil Karlton originally, sources differ.)
Hm, no. "A request without an Accept-Encoding header field implies that the user agent has no preferences regarding content-codings." vs. "An Accept-Encoding header field with a combined field-value that is empty implies that the user agent does not want any content-coding in response."
Yes, but in the real world, the outcome is always the same.
No. HTTP/1.1 introduced an IESG requirement that Host must always be sent and recipients must always reject with 400 if Host is not received. When I pointed out that not all URIs have a host, the response was that clients must send an empty Host field in that situation. It is a MUST requirement in Semantics section 5.4: This is ingrained in HTTP semantics and commonly used by non-browser HTTP implementations. There is only one example of an HTTP header field using "none" and that is Accept-Ranges. It isn't a good example for anything. To be clear, it would be a bug for an HTTP message processor to drop a received header field just because the value is empty. Nothing about SH will change that. What we are discussing is whether SH is capable of representing an empty list or an empty header field, both of which it is currently incapable of doing.
If that spec specifically says "empty field-value", a SH "empty" token will not be usable for Host: anyway, and we're back in the "generalizing from no example at all" again.
On Thu, Apr 11, 2019 at 03:53:15PM +0000, Poul-Henning Kamp wrote: After having thought a bit more about this, I now think that we don't need empty list elements in the end because lists are not strongly ordered, they're not arrays with a position so an empty element in a list brings no value. I'd further say that removing an element from a list must result in removing the list once the last one was removed as it is a set. So I think I'm fine with not dealing with empty lists. This then leaves us only with the empty header field alone, likely where a string or a token are expected. I don't see a reason not to support an empty string just like any other string. A header field is not just a value, it's a name+value combination. With a void value the name still serves as a boolean (present or not) and this is exactly how it's used when the field is made mandatory to distinguish between agents generations or as a signal to detect support for something. We've all filled administrative forms where the requested information was irrelevant but mandatory in the form and just left it void so that the person in charge deals with the particular case (on paper) or filled with whatever was accepted (on web forms) hoping it would just be ignored since irrelevant. For me the Accept and Host fields are exactly in this category. My concern is to be certain that a valid HTTP request can be transmitted over SH. And ironically, the only mandatory HTTP header ("Host") is also the one requiring the support for an empty value due to its mandatory nature. Willy
I have lost count of the number of packages I have received from USA where part of the address is "Denmark does not have states" :-)
Again, the purpose of SH is not to allow all possible HTTP header to be represented in the data model -- it's to solidify the 80% case and promote best (and interoperable) practice. Representing a boolean as an empty header field is not good practice.
See PR above.
I've put in the changes for the list-based header fields. It's less clear to me that it's desirable to default a missing or empty item to an empty string (strings and tokens) 0 (integers and floats), False (boolean) or a 0-length binary array (binary data); to me, those are all going to be application-specific defaults that are better specified explicitly. I'm happy to adjust the spec to make that more clear, if folks feel it already isn't.

Related discussion:
text: Dictionaries are ordered maps of key-value pairs, where the keys are short, textual strings and the values are items (item) or arrays of <del> items.  There can be one or more members, and keys are required to be unique. </del> <ins> items.  There can be zero or more members, and keys are required to be unique. </ins> Implementations MUST provide access to dictionaries both by index and by key.  Specifications MAY use either means of accessing the