jiebi/CodeConvo huggingface dataset:
In this particular case, it seems reasonable to me that the frame size error should halt further processing of the frame such that neither of the stream errors are even detected: after all, an incorrect frame size may indicate corruption (in which case the rest of the frame's data must also be suspect). But this is neither clear from the spec nor might other cases be so clear cut. I see three possibilities, with a reasonably strong leaning toward the first: Clarify that further processing of frames SHOULD (or MUST?) be halted upon encountering an error (I suspect this is what most implementations currently do, irrespective that the RFC says they "MUST" behave otherwise?). But then the order of processing becomes significant: is it right that this should be left as an implementation detail, or should it be specified? After all, the above frame might trigger a STREAMCLOSED stream error from one server and a FRAMESIZEERROR connection error from another, with very different consequences: how might these differences impact intermediaries, caching and retries? Better I think for the order to be specified. Define a precedence to errors, and mandate that that with highest precedence be sent (for example, connection errors might have higher precedence than stream errors). But then endpoints must continue processing frames they know to be erroneous simply to discover whether any higher precedence errors also exist (at least until an error of the highest possible precedence is encountered). Permit (require?) all errors arising from a single frame to be transmitted irrespective of any other errors the frame may also have triggered. Again, endpoints would then continue processing frames that they know to be erroneous—but the peer would then be fully informed of all problems in order that it is better able to recover.
I think the simplest explanation is that connection errors are those that suggest there's no way to (safely) continue attempting to parse anything. I suspect most implementations would find that this frame is not long enough to actually be a HEADERS frame and not attempt to parse it as a frame it cannot actually be. Thus, while a given implementation could possibly discover one of the stream errors first (thereby sending a RST_STREAM), once you discover the connection error, you're sending the GOAWAY and throwing the connection state away. The logical implementation is likely to find the connection error immediately and not read far enough to find the others.
I agree, but this is not specified.
Is it okay to leave as an implementation detail, even if most implementations will do the "logical thing"? Also this was just one example of a frame with multiple errors... others may not be so straightforward.
The question we should ask is: Is there any behavior that would change that would cause it to not interoperate here? I suspect that, regardless of how one interprets this, processing must stop, and the client may or may not receive an error, thanks to the vagaries of packet-loss. If this requires any clarification, I'd pick the path of asserting that any connection error stops any other processing.
Actually I also found error handling quite tricky to do, especially regarding when and what to send in GOAWAY frames. One really needs to think in terms of error severity to implement a form of funnel so that more severe errors can still be emitted after less serious ones. One example that quickly comes to mind is that we can send a GOAWAY frame to indicate a graceful shutdown of a connection yet this one is not an error and may be followed by other GOAWAY reporting protocol violations. I also faced an annoying situation related to TCP where you can't send a GOAWAY frame due to a buffer full situation, and closing after this causes a reset to be emitted due to pending incoming data. So we actually have to enter an input drain state in error situations, which further adds to the implementation complexity.
QUIC has that I think h2 would benefit from.
See for some proposed text to address this. We probably need to workshop it a bit.
Thanks!
This PR elaborates on the privacy analysis a bit. Describe why it is important (because we cannot delete items from the log) Remove the BCP 14 keyword (MUST NOT). Both the Privacy and Security Considerations sections should have the character of an analysis. They should be recommendations, or highlight gotchas to implementors. Anything binding should be in the main body of the RFC. Yes, this would need to be fixed in the Security Considerations, too. Expand the "Unless advertised by a Transparency Service" phrase into a paragraph about access control. This way it's more actionable. (Why would you advertise it? When would it be okay to advertise?)
Authors: (NAME NAME NAME NAME NAME I'm inclined to merge this as is. Any disagreements, or additional changes? We can iterate afterward, but seems like a good baseline to add
Yes. this looks pretty stable and uncontroversial. Thanks a ton, NAME
Agreed, but I don't see a way around having a "." be allowed there. If a system wants to further escape dots within labels if they know about them, that's OK...
The current text says names MUST be represented using a percent-encoded value instead. However, domain names can also contain and , so I think the escaping rule has to be a bit more generic than just escaping commas, as indicated in the current text.
Wire format DNS names can contain arbitrary octets. I'd suggest allowing characters in the set and %-encoding any other character. This is similar to the filtering that the glibc stub resolver performs on CNAME alias chains.
Reference RFC 1035 section 3.1 for the fact that these can be arbitrary Percent encode everything that it not alphanumeric or dash Mention in the security considerations that if you don't do this, you could inject newlines, etc.
Actually, sf-string already prohibits things like newlines:
The unreserved characters are: is a valid interior label character in a wire-format DNS name, which seems like it would allow spoofing a label boundary. Not sure how much of a problem this is.
Editorial, since this is merely restating locally what was previously incorporated by reference.
In draft 00 it says: >The layout and semantics of the frame payload are identical to those of the HTTP/2 frame defined in ]. The ORIGIN frame type is 0xc (decimal 12), as in HTTP/2. I think this is the text as it was, before we ran HTTP/3 and HTTP/2 through the process. I think there's some room for editorial improvements to avoid the guesswork. Not least because HTTP/2 Origin has an errata on the payload definition (URL). My suggestion is to just admit defeat and define the whole frame like we do in HTTP/3 or ext priorities, something like Then we can say something like the semantics of the HTTP/3 ORIGIN frame payload are the same as those of the HTTP/2 ORIGIN frame, modulo the differences already described (control streams and whatnot).
This just moves the reference to the canonical definition of what is valid up to the top of the section. That's the canonical text. This emphasizes the point that HTTP/2 is placing additional requirements on endpoints with respect to validation. This does not really in the sense that it leaves validation of DQUOTE and friends to the core semantics implementations. Note that any "MAY" requirement for rejecting a messaging can have the desired effect on those generating those messages: if an endpoint puts DQUOTE in a field name, that message will have little hope of being successfully handled.
My 2 cents: it seems that there's disagreement about whether the validity requirements from the core specs need to be repeated. Of course, DRY. Semantics currently says: The issue here is that H/2 actually extends that ABNF (without doing that as ABNF), which makes the link somewhat weak. Maybe formally (==ABNF) defining "h2-fieldname" as "lowercase-http-fieldname / h2-pseudofield" would be helpful.
So the proposal would be:
Julian, YES PLEASE for ABNF (yeah I know that is editorial, but it makes it so much more concise and exact). I think everybody agrees that a h2 impl MUST NOT generate invalid HTTP fields. The differences appear on the receiving side as the current document allows that h2 impls MAY accept some invalid HTTP fields. I've never understood the use-case for this? cheers
Well, AFAIU, we do not have a hard requirement on recipients in the base spec (and HTTP/1 either): So this seems consistent with what the base spec says.
I have no objection to adding ABNF to define H2’s fields. I would say that we should keep the prose, as the prose does a good job of expressing the why of our choices: for example, explaining the expectations around lowercase field names. However, I agree with NAME and my past self that we do not need add a “MUST validate” section.
Thanks Martin. I'm having a bit of difficulty parsing this sentence: "These checks are the minimum necessary to avoid accepting messages that might avoid security checks". Maybe "These checks are the minimum necessary to forward or process the message as an HTTP message" ? Also "A recipient can treat a message ..." why not "s/can/SHOULD" ?
I think that should be enough to solve it.
My objection to ABNF is the same as before: we risk that being mistaken as defining what is valid. Something that meets that ABNF is not valid. The point of this section is to define what the absolute minimum is with respect to any form of message handling. Our attempt to require more complete validation roundly failed, so this is where we end up. Willy, I've tried again with the wording. But I want to avoid normative statements regarding the core semantics documents. Those already define what a receiver might do. Those requirements aren't strict enough, at least in the "acting as a tunnel" case, for HTTP/2. The point of this text is to point to HTTP and note that these checks won't guarantee you a valid message, but to allow those implementations that do apply those checks the option of using HTTP/2 signaling (reset streams) to indicate the error condition. I don't think that needs a SHOULD, but I'm OK with adding it.
NAME I think this is a flawed approach. By only defining a subset of invalid fields we create a set of fields that are not valid HTTP but may be valid H2 fields (by merit of being not defined as minimally invalid - just describing it is convoluted). I have yet to see any use-case for a field to be valid in H2 but invalid in HTTP? has identified that the example we have frequently used for such a field (double quote in name) will be a concern if passed onto CGI scripting language. So we have no known uses for such fields and security concerns about at least some of them. So why not just define valid H2 fields as a proper subset of valid HTTP fields plus pseudo fields? We can avoid defining what is valid by referencing other specifications (which do use ABNF). As suggests a field should be valid in H2 if and only if it is: I.e. the set of valid H2 fields is a proper subset of valid HTTP fields plus pseudo fields.
That is what RFC 7540 did. That failed. We can revert all of these changes (aside from editorial tweaks) and go back to what the previous version said, but would that be any more successful?
I take it that you consider RFC7540 failed because of URL which ultimately references URL These indicate that there are cases of CTL characters in field values observed in the wild. The response has been to relax the validation on fields so much that now security problems have been created. I think this is because the minimal validation approach is essentially trying to be secure with a black list (in this case a list of known bad characters). It is far more secure to use a white list approach (i.e what is the minimal character set required to carry the HTTP semantic ). I don't really understand why no attempt was made to enforce the previous specification - I really hope it was not because of who was violating. But if enforcing the spec is seen as impossible, then rather than just leave the validity of fields somewhat undefined, surely a better approach would have been to make some explicit exclusions to allow just the needed control characters in field values. Or perhaps field names must be valid HTTP field names (or psuedo and lowercase), but h2 is more relaxed on field values? Deserting the field and not actually precisely defining what is valid just seams like an inviation to create more special cases and more unforseen security problems.
The real reason for this situation is that we all know that what is extracted from H2 then HPACK will need further processing and that this processing may already be fooled by the contents extracted from there. In addition, H2 being a standard multiplexed protocol on top of TCP is abused by some not really interested in the HTTP aspect of it (e.g. URL) , putting intermediaries at risk of breakage and giving them even more motivation for dropping essential controls and becoming insecure. While I would prefer a wording directly referencing the HTTP spec, like Martin I'm convinced that some will simply not even read it. So I think the proposed addition here can be welcome provided that it does not use the terms "valid" or anything else that might make implementations think "OK that's sufficient", and that presenting the root cause and the risk remains useful to convince those who are hesitating (or at least give them good arguments for not disabling the tests). Let's try again: It's not far from what Martin wrote in the last update, it just tries to better present the context and mention that this has to be done before HTTP parsing. And I'd like to use for treat as malformed.
No attempt was made to enforce the previous specification because the IETF and httpwg have no enforcement mechanism. What could have been done to enforce it? W.r.t. this patch, I agree with NAME that it seems that the conversation has come full-circle to where we were with RFC 7540. In that circle I think the wording here is closest to what I’d want: a clear list of specific rules with a note that these rules are a lower bound on processing, not an upper bound.
I agree that misunderstanding the safety of the current spec is a real problem, but I don't think avoiding the word "valid" will help. Many developers will naturally think that a HTTP/2 specification would only support valid HTTP fields. You can't fix this in the text because that assumption has already been made in existing software and will be made again without reading the text. Even those that do read the text might get lost in the fine distinctions we are trying to make here between a field that is valid and one that may not be invalid. What is a safe field? If the safety checks are not sufficient for consuming HTTP semantics, what are H2 fields safe for? What layers can consume them without extra validation? Why are consumers protected against CR, LF and ':' attacks but they are not safe from DQUOTE ones? Fields passing these safety checks cannot not be passed to any existing layers that are written with the assumption that they will only receive valid HTTP fields and new user will wrongly assume that H2 safety checks are at least as good as those in HTTP/1. RFC7540 said that an imple MUST be invalid HTTP fields as malformed, but some impls ignored that. So how about we say that impls SHOULD treat invalid HTTP fields as malformed and then define the specific circumstances in which consenting peers can exchange invalid HTTP fields, perhaps indicating as much with a SETTING or new psuedo header that is needed to specifically allow h2 framing to be used for non-HTTP semantics? Yeah I know some will say "you can't force those users of invalid HTTP fields to do something special". So instead we make all users of valid HTTP fields need to do extra validation else risk being insecure? Anyway, sorry for re-re-re-re-litigating.
My live-with criteria is that an implementation MAY treat invalid HTTP fields as malformed, so in that sense this PR is fine.... it just doesn't (and Martin says as much), so this is really just an editorial PR unrelated to .
Greg, I'm not suggesting that an H2 field is safe, but "safe to be parsed as HTTP". The main reason is that some basic HTTP components have been broken into pieces in H2 and need to be reassembled before being processed by HTTP. :scheme, :authority, :path are two such examples. In HTTP together they're called a URI. Cookie is another one. Some H2-specific issues may happen at the H2 layer that are impossible by definition in HTTP since HTTP/1. In my opinion this is the trouble that is being attempted to be addressed here. And yes it turns out that it diverges a bit from though it can address it if it mentions that consumed fields are subject to these checks. For sure, if we enforce strict validation of all H2 fields against HTTP ones it will address everything, except that we've already seen that implementors are not keen on switching between many specs at once and need some guidance. It's a delicate balance. Julian proposed to duplicate the ABNF there, I thought it was a good idea. And if we don't want to make it look normative we could write the minimum elements as a reminder.
But the spec currently as currently written and after this PR is not "safe to be parsed as HTTP". If that parsing is done by a scripting language then DQUOTE and other control characters that are current permitted by the spec may well not be safe. To take your example of a cookie header, how can you be sure that code which is currently safely the cookie header value (or set-cookie value) will be robust in the face of control characters within that value? Isn't the set of h2 fields that are "safe to be parsed as HTTP" a true subset of valid HTTP fields? What extra characters can we allow into fields that are not valid HTTP that we are sure will be parsed safely as valid HTTP by all existing layers currently running on top of H2 implementations? The intent of these changes have been to normalize the practise in the wild of h2 framing being used to carry control characters in fields for non-HTTP purposes.
But if we nomarlize that, we are putting in jeopardy all existing code written to HTTP semantics that may not even be aware of the version of HTTP carrying the fields that they are interpretting. Surely there is some other way to allow h2 framing to be used to carry non-HTTP streams without putting all existing HTTP code at risk? If we don't want developers abusing the current protocol, then give them a proper solution for their use-case rather than corrupt the existing primary use-case.
They should represent the exact same level of risk as having the same code parse HTTP/1. But actually that might in fact be what we're all seeking without precisely realizing it, which is making sure an H2 message can be at least reconstructed as an H1 message that is safe to parse. The rules on CR/LF precisely have always been a concern to all of us for this. Same as above. This is always difficult to say if we'd try to extend what HTTP permits. However we're certain that some will almost always be dangerous along a chain or cause parsing issues even inside implementations. Not exactly. This is actually the current situation probably because some consider that the effort needed to comply with the rules in semantics is too high for little value the this level. This is something I can understand. For example in haproxy we have a semantics layer which performs a lot of controls, deduplicates content-length, checks Host against etc. But at the lowest level it is difficult to enforce such checks when you're just converting header lists to internal messages. However we already have the NUL/CR/LF checks that allows the upper layer parser to proceed (the initial implementation used to build an H1 message but that's no longer the case). Thus maybe we should replace all this with an approach centered around this tricky HTTP/1 translation, which also does not infer how implementations should work internally nor suggest to put anything non-HTTP inside HEADERS frames: Just to be clear, I really do not want to see HEADERS frame being abused to carry non-HTTP because I know pretty well what it's like to be an intermediary developer whose product is pointed the finger at for breaking stuff when inserted in a working but non-compliant chain. This is also why I would like to see the SHOULD reject as malformed.
I like this paragraph and I'm totally OK with a spec that has conditions like "implementations which reassemble elements as an HTTP/1-like message before ...." However, it is not just HTTP/1 as there are many HTTP semantic layers that are unaware of the protocol version, so it needs to be any impl that represents streams as HTTP messages must do specific validation whilst the h2 "abusers" are free use streams to the limites of HPACK capabilities, so long as they don't represent the results as HTTP messages - either on the wire or to layers above. Furthermore an impl that is representing HTTP messages cannot just do the minimal safety checks you listed. It MUST validate the fields against the general syntax of HTTP fields. For example, if I provide a h2 implementation of the Java class, then my impl MUST validate the fields against HTTP syntax. But if instead, I use my h2 impl to implement a gRPC API, then I'm under no such obligation as I'm not representing HTTP messages. More over, it is only the protocol impl that can do this validation. Another example is that my server is used on PAAS platforms where the application was deployed decades ago and the source code is probably lost and the dev team all retired. The app cannot be redeployed and it certainly cannot be reviewed for how it parses the Cookie header. It's not sufficient for me to switch them from HTTP/1 to H2 and in the process expose those applications to non-valid HTTP headers in HTTP messages I deliver to them. If the h2 impl is not going to validate the fields are valid HTTP then who is? So how about something like:
The thing is, stacks that support multiple versions will certainly not implement the full header syntax in each version because that is exactly what ought to be done at the semantics layer and why the HTTP spec was split like this as well. We really have messaging (H1,H2,H3) and semantics. The transport explains how to extract elements from a byte stream and how to encode them into a byte stream. The semantics details their syntax and consistency. NUL/CR/LF purely come from the H1 world but H1 was HTTP before H2 existed, so it could be argued that they are legacy limitations that probably affects most stacks. The cookie header splitting is purely H2, and once recombined it must parse correctly according to the semantics definition.