jiebi/CodeConvo huggingface dataset:
Code change:
IANA is asked to enter the following into the "HTTP Content Coding Registry" registry (HTTP): <del> Name: br-d </del> <ins> Name: dcb </ins> Description: "Dictionary-Compressed Brotli" data format.

Related discussion:
text: If we were to do that, having a fixed length hash as part of the format would make things more complex.
Having a fixed-length hash (or fixed hash) as part of a content coding is perfectly fine. If there is a need to update hashes, it is easy to define a new content coding.
NAME the doesn't use skippable frames and uses the same custom header for Brotli and Zstandard (with different magic numbers). I can switch to using a skippable frame instead (which effectively just becomes an 8-byte magic number since the frame length is always the same) but I'm wondering if it makes sense and is worth adding 4 bytes. It won't help in creating the files so it's just for decode time and the main benefit you get is that you can use the existing zstd cli and libraries to decode the stream without skipping the header but those also won't verify the dictionary hash, they will just skip over it. That might not be a problem but part of the decode process will be to fail the request if the hashes don't match.
In talking to the Brotli team, it looks like Brotli already and validates it during decode to select the dictionary to use. It uses a "256-bit Highwayhash checksum" so we can't use the hash to lookup a dictionary indexed by SHA-256 but we can use it to guarantee the decompression doesn't use a different dictionary (and the existing libraries and cli tools already use it). NAME when you were concerned about the , was it for both lookup and validation or just validation? I'm just wondering if we can use the existing brotli streams as they already exist or if we should still add a header to support locating the dictionary by SHA-256 hash.
There are a few places in HTTP where the interpretation of a response depends on something in the request. Everyone one of those turns out to be awful for generic processing of responses. That's my main reasoning, so I'd say both: lookup first, then validation. That is, my expectation is not that the client has a singular candidate dictionary, or that it commits to one in particular. So in the original design, when you had the client pick one, that didn't seem like a great idea to me. For validation, Highwayhash (NIH much?)
Code change:
IANA is asked to enter the following into the "HTTP Content Coding Registry" registry (HTTP): <del> Name: br-d </del> <ins> Name: dcb </ins> Description: "Dictionary-Compressed Brotli" data format.

Related discussion:
text: doesn't appear to be pre-image resistant, so I'd be concerned if we were relying on the pre-image resistance properties of SHA-2. Can we be confident that this is not now subject to potential ployglot attacks if we relied on that hash alone? That is, could two clients think that they have the same resource, but do not?
I wouldn't be comfortable switching hashes given the wider and proven use of SHA-256 and agree on the lookup case to allow content-encoding with different and multiple dictionary negotiations. Looks like a separate header is still the cleanest so the main remaining question is if we use the same style of header for both or a skippable frame for Zstandard. I also need to update the brotli citations to point to the shared dictionary draft format instead of the original brotli format. On Sun, May 12, 2024 at 7:58 PM Martin Thomson NAME wrote: >
Sorry for the late comment. It seems to me that conveying this information in the content eases the integration with Signatures and Digest. It is not clear to me if there are still possible cases where the response does not contain all the information required for processing.
Thanks
Looks like you just need to update the PR summary to reflect this change.LGTMOverall, I like this. I want to flag for others that it doesn't use the skippable frame concept.
Code change:
5. <del> TODO </del> <ins> 5.1. Implementers should be aware that using the same client attestation across multiple authorization servers could result in correlation of the end user using the client instance through claim values (including the public key in the "cnf" claim).  Client deployments are therefore RECOMMENDED to use different client attestations across different authorization servers. </ins> 6. <del> 6.1. </del> <ins> TODO 7. 7.1. </ins> This section registers the value "client-assertion-type:jwt-key- attestation" in the IANA "OAuth URI" registry established by "An IETF

Related discussion:
text: As highlighted in the title this PR adds a privacy consideration around potential client instance tracking along with a recommendation for client deployments to mitigate such risk.
Code change:
2.1.1. If the value of an HTTP field is known by the application to be a <del> structured field (STRUCTURED-FIELDS), and the expected type of the structured field is known, the signer MAY include the "sf" parameter in the component identifier.  If this parameter is included with a component identifier, the HTTP field value MUST be serialized using the rules specified in STRUCTURED-FIELDS applicable to the type of the HTTP field.  Note that this process will replace any optional internal whitespace with a single space character, among other potential transformations of the value. </del> <ins> structured field type (as defined in STRUCTURED-FIELDS or its extensions or updates), and the expected type of the structured field is known, the signer MAY include the "sf" parameter in the component identifier.  If this parameter is included with a component identifier, the HTTP field value MUST be serialized using the formal serialization rules specified in STRUCTURED-FIELDS (or the applicable formal serialization section of its extensions or updates) applicable to the type of the HTTP field.  Note that this process will replace any optional internal whitespace with a single space character, among other potential transformations of the value. </ins> If multiple field values occur within a message, these values MUST be combined into a single List or Dictionary structure before serialization. <ins> If the application does not know the type of the field, or the application does not know how to serialize the type of the field, the use of this flag will produce an error.  As a consequence, the signer can only reliably sign fields using this flag when the verifier's system knows the type as well. </ins> For example, the following dictionary field is a valid serialization: If included in the signature base without parameters, its value would

Related discussion:
text: Can you suggest when that might be the case? Because I find it doubtful in the extreme that this will cause real-world problems.
I agree that this wouldn't cause real-world problems. I just also wish that our specs actually say what we think should happen, instead of requiring people to figure out. So in this case, I think we can solve this by saying "RFC 8941 or a revision obsoleting it".
Since NAME acknowledges that this doesnt affect interoperability and this is merely a "wish", NAME why don't you go ahead and incorporate your proposal so we can ship this spec?
NAME - please don't claim what I said something I did not say. My position remains: the spec should be clear about what it expects implementers to do. Right now it is not. If the agreement is "use RFC 8941 or a future revision", why don't we say that?
Proposed change: URL
Code change:
to HTTP/2, but are not redefined in this document. [RFC Editor: please remove this paragraph.]  IANA is requested to <del> update references in these registries to refer to this document.  The registration of the </del> <ins> update references to RFC 7540 in the following registries to refer to this document: Application-Layer Protocol Negotiation (ALPN) Protocol IDs, HTTP/2 Frame Type, HTTP/2 Settings, HTTP/2 Error Code, and HTTP Method Registry.  The registration of the </ins> method needs to be updated to refer to preface; all other section numbers have not changed.

Related discussion:
text: Includes changes from , as that hits the same paragraph.
why is c4d3528 included?
Hmm, poor rebase on my part I think. I made , you made the changes there, then I made a branch for this and then rebased on top of . It should go away when merged.
Code change:
this will cause parsing to fail, but it is not possible to reliably fail in all such circumstances. <del> The Display String type conveys a Unicode string without any form of sanitization.  Applications using these values need to perform their own checks on their content; for example, they might contain escape sequences, or NUL.  Mitigation strategies include escaping untrusted content before displaying it. </del> <ins> The Display String type can convey any possible Unicode code point without sanitization; for example, they might contain unassigned code points, control points (including NUL), or noncharacters.  Therefore, applications consuming Display Strings need to consider strategies such as filtering or escaping untrusted content before displaying it. See also UNICODE-SECURITY and I-D.draft-bray-unichars. </ins>

Related discussion:
text: Stupid language question: "all possible" or "any possible" ? All we say about the UTF-8 decoding is: Let unicodesequence be the result of decoding bytearray as a UTF-8 string {{UTF8}}. Fail parsing if decoding fails. But UTF-8 has enough "payload bits" to transfer up to 0x1FFFFF where Unicode is limited to 0x10FFFF. Should we include that in the caution somehow or is that (sufficiently) implicit in "if decoding fails" ? (This is obviously approaching bikeshedding, I'm fine with the current text.)
Changed to 'any'. I think it's implicit in 'if decoding fails'; is there a situation where it wouldn't?
I propose we ban from serialized , in order to avoid a large class of security issues (and to simplify implementations) in C/POSIX environments where is the string terminator. I can see three alternatives here. In the serialization of we add: if byte is %0x00, fail serializing. In the parsing of `Display String we add: if octet is zero, fail parsing. If either end uses a strict UTF-8 encoder or decoder, which do not allow "overlong byte sequences" (see below), this prevents from being transported with DisplayString. Java pioneered a handling of by specifying that it must be UTF-8 serialized as - a so-called "over-long UTF-8 byte sequence". We have presently not specified if the UTF-8 encoding of DisplayString is "strict" where such over-long byte sequences are illegal or "non-strict" where either all or a few over-long byte sequences are accepted. Over-long byte sequences in general are frowned upon, because they can be used to "obfuscate" UTF-8 strings, for instance by encoding a period as 0xc0 0xae to try to escape directory traversal checks. We could change the spec to say that the UTF-8 encoding/decoding must be loose (enough) to handle the Java-way, but I failed to identify a suitable normative reference which didnt bring in a lot of other UniCode/UTF-8 baggage we may not want.
Code change:
this will cause parsing to fail, but it is not possible to reliably fail in all such circumstances. <del> The Display String type conveys a Unicode string without any form of sanitization.  Applications using these values need to perform their own checks on their content; for example, they might contain escape sequences, or NUL.  Mitigation strategies include escaping untrusted content before displaying it. </del> <ins> The Display String type can convey any possible Unicode code point without sanitization; for example, they might contain unassigned code points, control points (including NUL), or noncharacters.  Therefore, applications consuming Display Strings need to consider strategies such as filtering or escaping untrusted content before displaying it. See also UNICODE-SECURITY and I-D.draft-bray-unichars. </ins>

Related discussion:
text: We can optimistically serialize any bytes we encounter in the encoded UTF-8 bytearray using the "java-trick" and leave it to the UTF-8 decoder to either reject or accept as it sees fit. In the serialization of we add: If byte is %x00 append "%c0%80" to encodedstring. In the parsing of we add: if octet is zero, fail parsing. I'm a big fan of clear text and simple solutions, and I cannot imagine why anybody would ever try to send through a for non-nefarious purposes, so I am 100% on board with simply banning bytes in the encoded UTF-8 byte-array. But it is a (tiny) loss of generality, and if we want to avoid that, we either make life difficult for implementers in C/POSIX based environments by rejecting this ticket, or we throw our lot with Java's handling of to a greater or lesser degree. Optimistically serializing , as specified in the third alternative is a way to do that without tying ourselves to Java's mast, but it introduces some wiggle-room which, depending on ones point of view, is either desirable (allowing the receiver to reject by using a strict UTF-8 decoder) or undesirable (making it anyone's guess if can be transported in a DisplayString or not.) But Java's handling is a bit of a hack, and allowing (some) over-long UTF-8 byte sequences introduces a class of security issues similar to the one this ticket tries to prevent, so all in all, I think we should just ban in serialized with the first alternative above.
IIUC, Display Strings aren't meant for sending byte arrays, therefore I do not think we need to define a special way for conveying NUL characters. Based on that, IMO the question boils down to if we should ban encoders from sending NUL. I do not have a strong opinion on if that should be forbidden, considering the fact that if the receiving application is suspectible to NUL characters decoders have to be careful in the handling of NUL characters regardless of the encoding requirement. Note that decoders are already expected to treat NUL characters specially if necessary, see Security Considerations. Maybe we can just add a reference from the section describing the decoding logic to the Security Considerations, and call it a day.
Code change:
this will cause parsing to fail, but it is not possible to reliably fail in all such circumstances. <del> The Display String type conveys a Unicode string without any form of sanitization.  Applications using these values need to perform their own checks on their content; for example, they might contain escape sequences, or NUL.  Mitigation strategies include escaping untrusted content before displaying it. </del> <ins> The Display String type can convey any possible Unicode code point without sanitization; for example, they might contain unassigned code points, control points (including NUL), or noncharacters.  Therefore, applications consuming Display Strings need to consider strategies such as filtering or escaping untrusted content before displaying it. See also UNICODE-SECURITY and I-D.draft-bray-unichars. </ins>

Related discussion:
text: One possible good point in favor of banning it is that we know that many configs will just act based on regex pattern search and might find some searched string after a %00 and consider they found what they were looking for, while the implementation will stop before when seeing the zero. I think we should simply say that "0x00" must never be serialized, hence "%00" must never appear in a DisplayString. This implicitly allows end users to write pretty simple protection rules (even regex-based) consisting in blocking the 3-char sequence "%00" in such fields. Encouraging the blocking on the receiver side can be efficient enough to discourage implementations from even trying to purposely send it.
There is a cost to banning it, which is that the Unicode string to DisplayString encoding function is no longer complete. While C strings can't represent embedded NULs, all other languages' string types can. Those languages would then have a harder time writing a DisplayString API, because now encoding can fail. So we're trading off a nuisance for C vs. a nuisance for everyone else. It is unclear to me what the second option buys us. The second option just means there are two ways to spell U+0000. A C decoder that uses a naive percent-decoded in-memory representation would still need to account for somehow. The third option would avoid that, but ultimately it just means the on-the-wire representation of the same U+0000 codepoint becomes more verbose. This, however, suggests a fourth option: keep the abstract data format and wire representation as-is. (All Unicode strings are allowed and is how you say U+0000.) Instead, non-normatively suggest that decoders that use a NUL-terminated in-memory representation MAY account for U+0000 by representing it as in memory.
I'm naively suspecting this could be used to attack some elements: fill such a field with as many %00 as you can and it gets doubled in its final representation (e.g. when passed to a next hop) :-/
By "this", do you mean the in-memory representation? I don't think that's a concern. If passing to a next hop is still over HTTP, the representation you send will continue to be because (under that proposal), that is the one and only representation of U+0000.
Code change:
this will cause parsing to fail, but it is not possible to reliably fail in all such circumstances. <del> The Display String type conveys a Unicode string without any form of sanitization.  Applications using these values need to perform their own checks on their content; for example, they might contain escape sequences, or NUL.  Mitigation strategies include escaping untrusted content before displaying it. </del> <ins> The Display String type can convey any possible Unicode code point without sanitization; for example, they might contain unassigned code points, control points (including NUL), or noncharacters.  Therefore, applications consuming Display Strings need to consider strategies such as filtering or escaping untrusted content before displaying it. See also UNICODE-SECURITY and I-D.draft-bray-unichars. </ins>

Related discussion:
text: It would continue to be incorrect to encode it as . If you're worried about a large in-memory representation, I will note that it costs the attacker three bytes ('%' '0' '0') of bandwidth to get you to represent something in two bytes ('\xc0' '\x80'). The attacker would get a better ratio be sending you a bunch of 'a's. But ultimately all this is just a question of what in-memory representation your program uses for U+0000. It just needs to unambiguously represents the possible state space. Indeed I might suggest, rather than doing these hacks, don't use NUL-terminated strings! Then this is moot. I certainly would not go with in anything I write.
I see no reason to single out U+0000 from other control characters. Either allow U+0000, or disallow all control characters. The overlong encoding hack will bring more issues than it will solve.
The only reason to single out U+0000 is that it is the string termination character in C and POSIX environments, and that is still a major cause of CVEs. I would not object to banning all the control characters, but that takes us into UniCode land which is full of dragons... That is a third way to get the indeterminacy of the "java-hack", but it does have the slight benefit that the implementor of the sf-bis parser may know if that over-long UTF8 sequence will work or not. However, if it does not, then nothing is gained with respect to predictability.
I'd prefer to stick with a caution here, rather than a prohibition. That means that endpoints (yes, whatever that means) can make decisions about what they tolerate, but we aren't constraining use. That is, the field can carry anything, but a resource might decide not to accept some values.
How about simply adding: "UniCode category 'Cc' (Control characters) are not permitted, unless the field definition unwisely explicitly permits them."
That's a field-specific parsing rule, which doesn't seem great from an API perspective.
Not if it goes in the "Defining New Structured Fields" section ? I think it is perfectly fair game to insist that fields which accept Control Characters are forced to specify that, and ban them by default.
Code change:
this will cause parsing to fail, but it is not possible to reliably fail in all such circumstances. <del> The Display String type conveys a Unicode string without any form of sanitization.  Applications using these values need to perform their own checks on their content; for example, they might contain escape sequences, or NUL.  Mitigation strategies include escaping untrusted content before displaying it. </del> <ins> The Display String type can convey any possible Unicode code point without sanitization; for example, they might contain unassigned code points, control points (including NUL), or noncharacters.  Therefore, applications consuming Display Strings need to consider strategies such as filtering or escaping untrusted content before displaying it. See also UNICODE-SECURITY and I-D.draft-bray-unichars. </ins>

Related discussion:
text: My hope is that adding such text will mean that no field will ever be defined to allow them.
Right, but the software handling that ban isn't a SF processor, it's the field-specific code. I think the most we could do would be something like how we handle field-specific constraint failures in 'defining new...', eg: But that creates a situation whereby the default is that there's nothing in the field's spec about control characters, the SF generic implementation passes them through, and the field-specific code is expected to know that it should reject them. Not great. So a better approach might be:
This is where we realize we have waded into the "no way to be a little bit pregnant with UniCode" sump, isn't it ? I still think that %00 is a valid security concern in C/POSIX environments, but I really do not want to open the UniCode can of worms. Maybe we should just drop this ticket and add a stern security warning that there is no such thing as "safe unicode strings" ?
I'd be good with a general unicode 'there be dragons' security considerations warning.