jiebi/CodeConvo huggingface dataset:
doesn't appear to be pre-image resistant, so I'd be concerned if we were relying on the pre-image resistance properties of SHA-2. Can we be confident that this is not now subject to potential ployglot attacks if we relied on that hash alone? That is, could two clients think that they have the same resource, but do not?
I wouldn't be comfortable switching hashes given the wider and proven use of SHA-256 and agree on the lookup case to allow content-encoding with different and multiple dictionary negotiations. Looks like a separate header is still the cleanest so the main remaining question is if we use the same style of header for both or a skippable frame for Zstandard. I also need to update the brotli citations to point to the shared dictionary draft format instead of the original brotli format. On Sun, May 12, 2024 at 7:58 PM Martin Thomson NAME wrote: >
Sorry for the late comment. It seems to me that conveying this information in the content eases the integration with Signatures and Digest. It is not clear to me if there are still possible cases where the response does not contain all the information required for processing.
Thanks
Looks like you just need to update the PR summary to reflect this change.LGTMOverall, I like this. I want to flag for others that it doesn't use the skippable frame concept.
Replaces
Thanks!
IMHO we need to be clear what the requirements of SF support are. The way the spec currently is written (referencing RFC 8941) means that SF-shaped field values using the new sf-date format are not supported. I don't think this is what we want, so unless we want to wait for sfbis to be ready, we'll need to add some consideration of how compat with SF extensions/revisions is supposed to work.
The intent of the current text is not to limit the use of the flag to only the currently defined formats. New formats are supported if the signer and verifier support the formats. The agreement of what format to use for a particular field is out of scope for this specification, and needs to be agreed upon by the signer and verifier outside of the signature process. This is true for all the current types and will remain true for any additional types defined in the future.
On a high level, this sounds right. As an implementer of a signature library, calling an SF library, what is your expectation what it'll do with sf-date once the spec is published? What if the sender want to sign an integer param in a dictionary, but the field instance contains another param using sf-date?
I would expect a library of that nature to publish its mapping between field names and SF types, and ideally allow that mapping to be edited by the caller of the library. Otherwise, how would a user of the library be able to provide a type mapping for a custom field that the library doesn't know about, let alone a new sf-type?
Understood. Different layers of libs. What I meant is a generic SF library, that just parses/serializes SF fields.
Then you need to know what you're parsing it in to. SF fields aren't entirely discoverable from the syntax alone, you need to know the type explicitly. I ran into this when implementing my own signature libraries. So the signature library needs to have that mapping, and if the underlying SF library doesn't support some new type, then you can't map to it. Once the dependencies are updated, you can provide that feature up the chain. It all seems pretty standard and simple to me, honestly.
How so? I still don't see the path to interop for any update of the structured field syntax, unless this spec at least provides some hints.
You need to know the top-level type (Dictionary, List, or Item) for each field. The consequence of that (AIUI) is that a recipient wishing to validate a signature that uses will need to understand the top-level type of each 'd field in the signature. It'll also need to support the permissable types in those fields -- e.g., if a field allows Date, the recipient needs to be able to handle that. As a result, it won't be possible to reliably verify a signature if your processor doesn't have knowledge of the top-level type, or if it doesn't have support for an extension type in use in one of those fields. However, there is a chance of verifying it still, if the SF-produced form is sent on the wire and not changed at all in transit, and the processor just ignores processing (and in some ways, this is worse than reliably failing). I think this all can be reasoned out by reading the specs appropriately. Documenting it as a tripping hazard would help users avoid interoperability issues, but this specification is already long and dense, and if we also make it a user guide, it will risk becoming unusable. So, speaking personally -- if the above summary is correct, I could see adding a sentence or two to section 2.1.1 at the most; I could also see leaving it to a separate profile/user guide.
[ chair hat on ] This issue is the last thing holding the Signature spec going to IETF LC. NAME NAME please respond to above, particularly: whether such text is necessary whether such text will address the issue when such text, if necesary, would need to be incorporated (before/after LC). Thanks,
I believe it would be good to have such text; and that it should be in the spec, not in a guide that is unlikely to materialize. It should be incorporated before LC (I hope you mean a new WGLC here...?). Given the amount of changes, we really should spend some time reviewing all that changed, and to verify that the open issues have been addressed properly.
NAME can you propose something? I don't see any reason to rerun WGLC, based upon the current PRs; there are some changes (especially in one or two), but they're based on AD review and can be discussed during IETF LC.
cc NAME
I would prefer to see your proposal :-). The RFC diff since the last WG version has 143 change blocks: URL We also currently have 17 open issues/PRs: URL IMHO we need to get these closed (can be rejecting or fixing), and also have a look at those that we raised since -13 and closed in the meantime. I don't think we do the community a service by sending this to IETF LC right now.
NAME you are grossly misrepresenting the state of the document. For one, the last WG version, and the one that went to the AD, is -15, not 13, so your diff above is misleading at best. The changes since then have largely not been substantive in nature, and all have been made publicly with the view and input of the WG. The open issues are nearly all addressed by open PRs, which I am waiting for AD feedback before merging (I'm not sure if the AD wants to review the PR text or the merged draft that results). What remains is this issue, as NAME pointed out above. To answer NAME 's question about how to resolve this: I think pointing out as a potential tripping hazard is not necessary but would not be harmful. The precondition to know the type is already in the text as the first sentence about the flag in section 2.1.1: Additionally, the lack of knowledge of the type of the field is already called out explicitly as a required error condition in section 2.5: I would propose we simply add a note to 2.1.1 of the form:
Well, it's hard to say what remains to be done until an issue is actually closed. We're all volunteers here, so it's good to make things as easy as possible to reviewers. So, trying to rephrase: let's get all open issues closed (one way or the other), and let the people who raised the issues give an opportunity to provide feedback before sending this back to the IESG. (I, for once, am not convinced that I agree with the proposed solutions, but due to the type of diffs (long lines) it's a bit tricky to understand what actually changed) works for the sending side; but what is a recipient supposed to do when the sf flag is present and the field value does not parse due to sender and receiver disagreeing about the SF version?
Also, that only affects the toplevel type of the structured field. It does not address the case where an extension parameter is added is an sf-date (or any future new type).
NAME until you propose text, it will be hard to say what kind of text you think would close this issue.
Except for this issue and two others recently raised by you (which will be considered as IETF LC issues), all of the issues and PRs you mention were generated by the AD review before IETF LC; this is a normal part of the process that does not require going back to WGLC unless the AD asks for WG review, which . NAME made a reasonable proposal. If you disagree with it, make your own.
If the proposal was: Then this isn't sufficient as it only applies to the toplevel type (see URL). NAME - I believe you are in the best position to make a proposal; but that said; I can try something later today.
So, we have in 2.1.1: This implies that the "sf" parameter can only be used on fields that conform to RFC 8941, no extensions allowed. I would suggest clarifying that, at the end of 2.1.1, with: "Note that the normative requirement to use the SF serialization algorithm precludes any use of future extensions of the structured field syntax. Support for future versions of SF will require an update to this specification." I believe this matches what NAME suggested.
Except that your proposed text is incorrect. The implication you are drawing is not true, nor what others have taken from that text. The use of the parameter does not preclude extensions to RFC 8941, as has been discussed at length here and on the mailing list. Extensions to sf types would not require changes to the signatures spec. I propose closing this issue without change to the signatures spec.
Sorry. Mark asked me to propose text, and that's what I did. The text has a normative requirement to use an RFC8941 serializer, so the proposed text is correct. That we disagree here is proof that a clarification is needed. If you feel that that p.o.v.
is too restrictive, we'll have to somehow relax the requirement on the serializer (for instance, by stating that the serializer needs to implement 8941 or a successor RFC - that is an RFC that obsoletes 8941). FWIW, I agree that this would be good. But it is not what the spec says.
No, we reference the current document and if that document is updated or obsoleted by something else after publication then you as an implementer go and follow that. That's how updating RFC's already work. All the specs and systems that implemented RFC2616 didn't suddenly break and become invalid when 7230 or later 9110 came out. Here, It's up to the updaters of 8941 to make sure there's A serialization section. Your view of specification dependency is not at all in line with reality.
I see where you're coming from; I'd appreciate if you would do the effort to understand where I'm coming from. The difference here is that any spec that extends 8941 with new types by definition is not backwards compatible. This is because 8941 has no extension points, has drastic error handling and thus requires a wholesale spec update for new things. This is very different from most other RFCs that we're using here. If there is agreement about what we want, why don't we just say it?
To illustrate further: RFC8941-vs-8941bis is like XML1.0-vs-1.1 or HTTP1.1-vs-2.0, not like RFC2616-vs-RFC9112.
That's a very specific definition of 'not backwards compatible' and one that I suspect many would be surprised by. Just because there are no explicitly nominated extension points doesn't mean you can't evolve the specification in a backwards-compatible fashion; the key requirement is that valid inputs per the 'old' version are also valid inputs per the 'new' revision. That is the case for sf-bis. It is true that an invalid input for RFC 8941 will now become valid; specifically, a type value beginning with 'NAME used to generate an error, and now will not (provided that the following characters are appropriate). The reasonable question to ask at this point is whether it's realistic / probable that an application will be depending upon that error being thrown to operate.
Can you suggest when that might be the case? Because I find it doubtful in the extreme that this will cause real-world problems.
I agree that this wouldn't cause real-world problems. I just also wish that our specs actually say what we think should happen, instead of requiring people to figure out. So in this case, I think we can solve this by saying "RFC 8941 or a revision obsoleting it".
Since NAME acknowledges that this doesnt affect interoperability and this is merely a "wish", NAME why don't you go ahead and incorporate your proposal so we can ship this spec?
NAME - please don't claim what I said something I did not say. My position remains: the spec should be clear about what it expects implementers to do. Right now it is not. If the agreement is "use RFC 8941 or a future revision", why don't we say that?
Proposed change: URL
Stupid language question: "all possible" or "any possible" ? All we say about the UTF-8 decoding is: Let unicodesequence be the result of decoding bytearray as a UTF-8 string {{UTF8}}. Fail parsing if decoding fails. But UTF-8 has enough "payload bits" to transfer up to 0x1FFFFF where Unicode is limited to 0x10FFFF. Should we include that in the caution somehow or is that (sufficiently) implicit in "if decoding fails" ? (This is obviously approaching bikeshedding, I'm fine with the current text.)
Changed to 'any'. I think it's implicit in 'if decoding fails'; is there a situation where it wouldn't?
I propose we ban from serialized , in order to avoid a large class of security issues (and to simplify implementations) in C/POSIX environments where is the string terminator. I can see three alternatives here. In the serialization of we add: if byte is %0x00, fail serializing. In the parsing of `Display String we add: if octet is zero, fail parsing. If either end uses a strict UTF-8 encoder or decoder, which do not allow "overlong byte sequences" (see below), this prevents from being transported with DisplayString. Java pioneered a handling of by specifying that it must be UTF-8 serialized as - a so-called "over-long UTF-8 byte sequence". We have presently not specified if the UTF-8 encoding of DisplayString is "strict" where such over-long byte sequences are illegal or "non-strict" where either all or a few over-long byte sequences are accepted. Over-long byte sequences in general are frowned upon, because they can be used to "obfuscate" UTF-8 strings, for instance by encoding a period as 0xc0 0xae to try to escape directory traversal checks. We could change the spec to say that the UTF-8 encoding/decoding must be loose (enough) to handle the Java-way, but I failed to identify a suitable normative reference which didnt bring in a lot of other UniCode/UTF-8 baggage we may not want.
We can optimistically serialize any bytes we encounter in the encoded UTF-8 bytearray using the "java-trick" and leave it to the UTF-8 decoder to either reject or accept as it sees fit. In the serialization of we add: If byte is %x00 append "%c0%80" to encodedstring. In the parsing of we add: if octet is zero, fail parsing. I'm a big fan of clear text and simple solutions, and I cannot imagine why anybody would ever try to send through a for non-nefarious purposes, so I am 100% on board with simply banning bytes in the encoded UTF-8 byte-array. But it is a (tiny) loss of generality, and if we want to avoid that, we either make life difficult for implementers in C/POSIX based environments by rejecting this ticket, or we throw our lot with Java's handling of to a greater or lesser degree. Optimistically serializing , as specified in the third alternative is a way to do that without tying ourselves to Java's mast, but it introduces some wiggle-room which, depending on ones point of view, is either desirable (allowing the receiver to reject by using a strict UTF-8 decoder) or undesirable (making it anyone's guess if can be transported in a DisplayString or not.) But Java's handling is a bit of a hack, and allowing (some) over-long UTF-8 byte sequences introduces a class of security issues similar to the one this ticket tries to prevent, so all in all, I think we should just ban in serialized with the first alternative above.
IIUC, Display Strings aren't meant for sending byte arrays, therefore I do not think we need to define a special way for conveying NUL characters. Based on that, IMO the question boils down to if we should ban encoders from sending NUL. I do not have a strong opinion on if that should be forbidden, considering the fact that if the receiving application is suspectible to NUL characters decoders have to be careful in the handling of NUL characters regardless of the encoding requirement. Note that decoders are already expected to treat NUL characters specially if necessary, see Security Considerations. Maybe we can just add a reference from the section describing the decoding logic to the Security Considerations, and call it a day.
One possible good point in favor of banning it is that we know that many configs will just act based on regex pattern search and might find some searched string after a %00 and consider they found what they were looking for, while the implementation will stop before when seeing the zero. I think we should simply say that "0x00" must never be serialized, hence "%00" must never appear in a DisplayString. This implicitly allows end users to write pretty simple protection rules (even regex-based) consisting in blocking the 3-char sequence "%00" in such fields. Encouraging the blocking on the receiver side can be efficient enough to discourage implementations from even trying to purposely send it.
There is a cost to banning it, which is that the Unicode string to DisplayString encoding function is no longer complete. While C strings can't represent embedded NULs, all other languages' string types can. Those languages would then have a harder time writing a DisplayString API, because now encoding can fail. So we're trading off a nuisance for C vs. a nuisance for everyone else. It is unclear to me what the second option buys us. The second option just means there are two ways to spell U+0000. A C decoder that uses a naive percent-decoded in-memory representation would still need to account for somehow. The third option would avoid that, but ultimately it just means the on-the-wire representation of the same U+0000 codepoint becomes more verbose. This, however, suggests a fourth option: keep the abstract data format and wire representation as-is. (All Unicode strings are allowed and is how you say U+0000.) Instead, non-normatively suggest that decoders that use a NUL-terminated in-memory representation MAY account for U+0000 by representing it as in memory.
I'm naively suspecting this could be used to attack some elements: fill such a field with as many %00 as you can and it gets doubled in its final representation (e.g. when passed to a next hop) :-/
By "this", do you mean the in-memory representation? I don't think that's a concern. If passing to a next hop is still over HTTP, the representation you send will continue to be because (under that proposal), that is the one and only representation of U+0000.
It would continue to be incorrect to encode it as . If you're worried about a large in-memory representation, I will note that it costs the attacker three bytes ('%' '0' '0') of bandwidth to get you to represent something in two bytes ('\xc0' '\x80'). The attacker would get a better ratio be sending you a bunch of 'a's. But ultimately all this is just a question of what in-memory representation your program uses for U+0000. It just needs to unambiguously represents the possible state space. Indeed I might suggest, rather than doing these hacks, don't use NUL-terminated strings! Then this is moot. I certainly would not go with in anything I write.
I see no reason to single out U+0000 from other control characters. Either allow U+0000, or disallow all control characters. The overlong encoding hack will bring more issues than it will solve.
The only reason to single out U+0000 is that it is the string termination character in C and POSIX environments, and that is still a major cause of CVEs. I would not object to banning all the control characters, but that takes us into UniCode land which is full of dragons... That is a third way to get the indeterminacy of the "java-hack", but it does have the slight benefit that the implementor of the sf-bis parser may know if that over-long UTF8 sequence will work or not. However, if it does not, then nothing is gained with respect to predictability.
I'd prefer to stick with a caution here, rather than a prohibition. That means that endpoints (yes, whatever that means) can make decisions about what they tolerate, but we aren't constraining use. That is, the field can carry anything, but a resource might decide not to accept some values.
How about simply adding: "UniCode category 'Cc' (Control characters) are not permitted, unless the field definition unwisely explicitly permits them."
That's a field-specific parsing rule, which doesn't seem great from an API perspective.
Not if it goes in the "Defining New Structured Fields" section ? I think it is perfectly fair game to insist that fields which accept Control Characters are forced to specify that, and ban them by default.
My hope is that adding such text will mean that no field will ever be defined to allow them.
Right, but the software handling that ban isn't a SF processor, it's the field-specific code. I think the most we could do would be something like how we handle field-specific constraint failures in 'defining new...', eg: But that creates a situation whereby the default is that there's nothing in the field's spec about control characters, the SF generic implementation passes them through, and the field-specific code is expected to know that it should reject them. Not great. So a better approach might be:
This is where we realize we have waded into the "no way to be a little bit pregnant with UniCode" sump, isn't it ? I still think that %00 is a valid security concern in C/POSIX environments, but I really do not want to open the UniCode can of worms. Maybe we should just drop this ticket and add a stern security warning that there is no such thing as "safe unicode strings" ?
I'd be good with a general unicode 'there be dragons' security considerations warning.