jiebi/CodeConvo huggingface dataset:
I think that this requires only that the text acknowledge this uncertainty and note that clients could decide to provide authentication information on every request made to the origin, without knowledge of the extent of the protection space. It might also note that particular authentication schemes might define mechanisms that allow clients to decide where to use credentials. RFC 7616 defines , which allows for scoping; RFC 7617 has a section on .
re the first point: we are defining the framework here; whether a scheme uses realm, and how exactly it is used, is out of scope here. What we do is reserve the parameter name "realm", and say what it is for. Could an auth scheme use it in credentials? I believe the answer is "yes".
I think both aspects are up to the authentication scheme in question. Martin, do you feel strongly about this, or can we close?
As you say, it's kinda up for grabs. The main spec isn't very satisfactory though in that it doesn't specify, but doesn't really admit to why that is. Maybe if I can be given a chance to think of some text for this and you can see if you like it.
This is a partial revert of 5469d51 from PR
Talked with NAME a moment ago. I think there are some additional changes that will be necessary, but I don't object to the idea of this revert. Seems like a mistake in the original PR.
Happy to amend the PR if you can give me an idea of what additional changes are required.
I wrote up a change that should cover all the points we want to make. Namely, we still want UAs to accept that larger character set while encouraging server to use the more limited character set.
This looks reasonable to me, thanks to you both.
Just state it. No need to justify it so (especially as this invites questions about the claimed benefits). This is less than Julian wanted from , but I think that it is important to include a requirement like this very clearly and it isn't really said anywhere else. At least not directly.
converted to lowercase when constructing an HTTP/2 message. I think this is somewhat misleading, it just provides the motivation why the lowercase format was introduced initially. I would just remove the sentence and potentially insert a note about lower-casing into the field name validity statements.
Agreed, I don't think the spec needs to defend itself on this choice.
(Contributing text as an individual, chair hat off) Here's some initial proposed text based to Please feel free to tweak my proposal.
Very nice. Looks good to me. Thanks for this!
The Introduction correctly points out the following: The above text says guidelines for specifications are important, but such guidelines only make sense if there is a specification in the first place. So I wonder if it's worth saying a few words here about why it's useful to specify/standardize congestion control algorithms. A few thoughts on the benefits of specifying congestion control algorithms: Help get a shared understanding between implementers, operators, and other interested parties of how the algorithm works and how it is expected to behave (or explicit discussion of parameters and such which can lead to differences in behavior) As a result of the shared understanding: Make it easier for anyone, not just the initial proponents or main implementers, to point out current issues/limitations or suggest improvements of a spec that is still in development, and make it easier to converge such changes to a consensus behavior There may be more benefits. Though I wonder how to best capture the nuance of "Some documentation or artifacts are better than nothing, but an actual spec is better than just a snapshot for documentation or artifacts". Do people think it'll be useful to add something along these lines to the document?
I agree it would be great to add this to the doc, and agree that those two bullet points would be good to integrate. Another more specific benefit that may make sense to add: due to the constraints of open source licenses, some implementors have noted that in some cases they are unable to read open source reference implementations; in such cases it is important to have a detailed spec of the algorithm that is unencumbered by open source license constraints.
I can see this would be of value to someone considering submitting a proposal. It also may help them to justify the effort in writing this down. If we can find words that work, I'd like to consider adding something on this.
Guidelines on what to consider are good for anyone (those who design, those who implement and those who document). The only place where guidelines could be used for enforcing something is where people wish to have congestion control algorithms published through the IETF. But that doesn't make them useless in other parts of the process. (My friend who does CC for realtime media is developing his work through iterations of code, not iterations of documentation. Guidelines would still be useful for him.)
This PR clarifies a few things around the Upload-Limit field from : The last upload append request may now be smaller than the minimum append size specified in . For example, if a 510 MB upload is split into 100 MB requests, the last request can just be 10 MB. It clarified how a client can indicate the entire upload length when creating a new upload via the Content-Length header. The PR also mentions that no resumable upload resource may be created if the minimum size is not met, but the server might still accept and process the data (although without offering resumability) The Upload-Limit header is added to informational response in an example
Introduce the Cross-Device Consent Grant label (see )
e.g. Malicious Context Change Authorization Request or something like that.
Ideas from Pieter's slides: Illicit Consent Grant Attack? Describes outcome, not the mechanism Attacker-in-the-Middle Attack? Describe attacker capability, but both too broad and too narrow Authorization Context Manipulation Attack? Describes the mechanism Authorization Context Manipulation Exploit Describe mechanism, hints that protocol functions as expected. What about "Phishing for Consent"? Describes the mechanim which depends to some extend on social engineering (phishing) Describes the outcome (consent) Other names shared via e-mail by George Fletcher: Zishing Azishing "Cross-Device Authorization Exploit" "Social Engineering Token Theft" "Authorization Flow Manipulation Exploit" FlowJack AuthJack TokenJack Context Manipulation Authorization Exploit Permitphishing, Authishing
"Cross-Device Consent Phishing"?
Some references on Consent Phishing: URL URL URL
CDCP - Cross Device Consent Phishing - the abbreviation works well.
Two editorial nits, the rest looks good to me! Thank you!
I also suggest closing here : the main reason for getting data without a request is an error reported by the server before closing (e.g. 408). Another reason may be a server implementation issue (extra CRLF, inconsistent multiple content-length, inconsistent content-length+transfer-encoding, user-fed incorrect error message). In all these cases the message delimitation is wrong and unrecoverable. Thus the client trying to "fix" it will in the best case be useless (close following), and in the worst case have nasty side effects. In haproxy the only case we try to address is the extra CRLF that a few old servers send slightly after data. If extra bytes are sent immediately after data, we close. If the extra CRLF happens slightly later, we find it while parsing the next response and the CRLF is silently skipped.
Everyone OK with this now?
Well, like Roy, I'd even prefer a MUST close, but I can live with a SHOULD.
It doesn't seem to account for the extra CRLF case. Maybe just add "... unless the data is limited to CRLF"?
CR and CRLF?
I was thinking the same but hesitating. BTW NAME it would be LF and CRLF. But quite frankly we could stick to CRLF only since it's the only known bogus pattern widely deployed in field.
updated.
Closes URL I am not super sure about the wording here, but this is a start. The sentences seem quite long and maybe the rules could also be expressed in fewer words.
I wonder if it is also possible to expose this final upload size in the response to Offset Retrieving Procedures. I don't think we can set in the HEAD responses because that would indicate that the server already has this amount of bytes, which is not correct.
If the client acknowledged its final upload size at any point ( + ), it should still know it in the future, so I don't think we need a way for the server to echo it in the offset retrieving response. If the server disagrees with the client, then someone's data representation has changed, and we shouldn't continue the upload anyway.
Looks good. Wondering if these could be reduced to SHOULDs though. It wouldn't really affect interop if the server didn't implement this.
Fair point, let's leave such functionality out of the draft. While it is true that clients usually won't really notice if the server does or does not support this, I think we can be stricter here and require this additional safety catch here. It will help to create a more standard landscape of resumable upload servers. Unless there is something against requiring this, we can keep it a MUST IMO.
I just updated this PR to use the Upload-Complete header as agreed on in
The arguments for SHOULD are: This check requires additional code and complexity. MUSTs are situations that a minimal implementation has to handle anyway and we are just strict about how to handle them. For people using chunked uploads, this check is only useful if the last chunk is interrupted which is sort of an edge case. Maybe we can generalize the check if we adopt ranged PATCH. I don't have a strong opinion on this, so either way is OK with me.
When the server receives a series of appends: If the offset+length is inconsistent, the server should reject the upload.
To clarify; Is what you are saying here that the provided during the upload creation procedure should act as the total size of the file, similar to Upload-Length in tus1x?
I am not sure if we want this. I ran into this exact problem because your NIOResumableUpload implementation uses this logic (see URL). We currently have the header to indicate the completeness of the upload.
Based on this information, the server can determine if an upload is complete or not without relying on . Adding to the upload logic would mean that we have two indicators for upload completeness, which could potentially disagree with each other. Personally, I think we should not use to determine the upload's completeness and only determine it based on whether the client set in a successful request. For each Upload Creation Procedure or Upload Appending Procedure, the should be set to the number of bytes that the client wants to send in this individual procedure. For example, if you want to transmit 100 bytes in the first POST request, set (or not if you use ). If you want to transmit 200 bytes in the following PATCH request, set . If you want to finish the upload then, set and the upload is considered complete with its 300 bytes of content.
The issue in NIOResumableUpload was a bug, not how this is supposed to work. Offset + Content-Length = final upload size when the upload is complete, and if the client has told the server about the final size, it shouldn't be allowed to change its mind.
Echoing what Guoye said, should not determine the upload's completeness, but if an request has a , then the server can expect and enforce the final upload size based on that . This prevents a client from appending more bytes to their upload than the server anticipated. Ex. Client starts an upload , Server expects a final upload size of 100 bytes Upload is interrupted, offset-retrieving, server sends Client now tries to append an extra ~10GB Client sends , , Server can reject this request
I see, thanks for the explanation. That makes sense. Basically, in conjunction with let the server know the final upload size, which should not be changed
How does that work if the last append has in conjunction with . The Content-Length needs to match the number of bytes in the append message.
Final size = upload-offset + content-length if the upload is complete
So the first ever request with Upload-Incomplete: ?0 and content-length sets the expected final size. Every other request must have a content-length less than that value?
Yeah, once the final length is determined, it cannot be changed by subsequent requests.
OK I think that makes sense but it's not obvious from the spec. We should try and make it more accessible with some new or different text.
-- URL This doesn't seem to be enforceable in any way. This isn't really talking about whether the syntax is valid, but about the semantics, so it's easy to disavow knowledge of falsehood (my browser isn't necessary aware that me typing "the sky is pink" is false). I think that it might be better to simply observe that it is contrary to function - and possibly interests - of a sender to generate protocol elements that convey semantics that are incorrect or contrary to the interests of the sender. Normative language implies interoperability failure if it is ever violated, and I don't think that is the case here.
It is enforceable. If a sender communicates a syntactically valid construct that is deliberately misleading, then the recipient's behavior in response will be equally confusing to all parties. We saw exactly that both in the deployment of HTTP/1.1 and in the early deployment of DNT. Both resulted in interop failure, and both failures were only corrected after the standards were enforced by recipients refusing to accept invalid signals.
We might be talking past each other. NAME - could you give us a concrete example that this text is trying to rule out?
Or, for example, P3P, where people deliberately set false and misleading policies so that browsers wouldn't complain. Roy, it's a nice thought, but this is the kind of regulation that's done by law, not architecture.
Both laws and RFCs are defining standards of conduct. The difference with laws is sovereign enforcement, not social versus technical. In any case, this has been true of HTTP implementation (Apache httpd, for example) since 1996. A server will check for compliance based on User-Agent version match and ignore protocol signals that are known to be false or at least unreliably implemented by that sender. Browsers do the same for some servers that deliver default Content-Type settings by mistake. We consider the workarounds to be standards-compliant because of this MUST NOT. And, as I said, we enforce that ALL OF THE TIME.
Discussed at Feb 21 interim; add context to help readers understand why this requirement is here / important. NAME to PR.
Context... ? Then add an example, probably a fictitious one out of deference for the sensitivities in all the truly good examples.
This works very nicely. Thanks Roy.
Starting to unpick where equal access is important and where harm for others becomes important. If merged, this
Issue possibly related to , but more focussed on refocus away from a fairness evaluation. When a transport uses a path to send packets (i.e. a flow), this can impact other Internet flows (possibly from or to other endpoints) that share the capacity of any common network device or link (i.e., are multiplexed) along the path. In turn, this can result in variation in capacity, loss, or additional latency. Transports and network layer scheduling can be designed to offer fairness between sharing flows. This comes with both strengths and weaknesses - it can reduce collateral damage to other flows; but it also can motivate the use of additional flows to gain additional network resource. Transport proposals need to be evaluate and avoid inducing flow starvation to the other flows that share resources along the path they use. Transports proposals need to be evaluate and ensure they treat a loss of all feedback (e.g., expiry of a retransmission time out) as an indication of persistent congestion.
Gorry will start with a PR that defines the terms; then we can talk about what to emphasize.
Also is related
One of the guidelines is "Fairness within the Alternate Congestion Control Algorithm". This a great consideration, but behind that are quite a few problems. The classic fairness evaluation derives from the equation about asymptotic behavior drawn by Matt Mathis, which end up requiring a non linear response to the frequency of a control signal such as packet loss or ECN marks. There are of course two problems. First, time scale, because many connections don't last long enough to approach asymptotic behavior. Second, scaling, as discussed for example in RFC3649. If an algorithm requires ever increasing spacing between control events to sustain larger data rates, then the control loop becomes very long, or, in the case of packet losses very noisy as "true" control events get intermixed with random losses. In contrast, algorithms with linear response scale very well, but do not naturally provide fairness. Which is kind of a bummer, and also a research topic. I think the Bis draft will have to recognize that tension somehow, and encourage experimentation.
The Guidelines section asks developers of new congestion control algorithms to discuss the "Impact on Standard TCP, SCTP ], and DCCP ]."
Maybe I am overly sensitive, but this way of discussing the issue feels dated, from a time when transport and congestion control where specified jointly. I think that ceased to be the case many years ago, for example with implementations of TCP providing choices between multiple congestion algorithms. (Indeed, DCCP itself provides a choice between two algorithms.) The Bis draft should probably present the issue differently.
Speaking of fairness, I also wonder if there is an accurate definition for it. There has been activity on the topic at the IETF (on in academia) : URL URL URL should this be discussed in the draft ?
Discussed at IETF 117: Matt: Replace fairness with freedom from starvation. Fairness only makes sense at low bandwidth. Christian: Not focus entirely at fairness, non-starvation goal is good. Gorry: Suggestion to put it in other draft, but not in this one Overall, get rid of it all together for now
In academia a next metric is to look at harm rather than fairness: URL
That's a pretty good reference, Mirja!
The reality is that Cubic has a horrific effect on the Internet: it is pretty efficient at building queues and making life terrible for others. The classic example is the parent busy on a video conference from home, and then seeing the perceived quality go down sharply because the teenager started downloading a movie... Which means I would take "fairness with Cubic" with a huge grain of salt. As is, yes, let it run and get some bandwidth, but make sure that it gets seriously penalized, so there is an incentive to replace it by something better...
A more serious problem is harm to tiny transactions using any protocol. They have to rely on timeouts for loss recovery. I suggest replacing this with absolute upper bounds on loss rates. E..g. see: URL Likewise for excessive standing queues, and to a lessor extent excessive CE marks.
Modulo the comments I've already posted, LGTMThe last changes by NAME probably mitigate my issues. I think we will have to come back to definition of harm and to the "self compete" part, but this is good enough for the next iteration of the draft.
Closes URL
I suspect a title like "HTTP Resumable Uploads" or "Resumable Uploads for HTTP" would be more clear -- not referring to HTTP will pull in readers looking for other things, and calling it a protocol reinforces that it's separate, when we're trying to make it an extension to an existing protocol.
Good idea to include HTTP in the title! I updated the PR.
Great suggestions, Lucas! Thank you very much!
LGTM, thanks for picking it the pen.
The title of the resumable uploads draft starts with "tus", which is a little weird as the inclusion is entirely unexplained. If the intent is to acknowledge something, might I suggest the Acknowledg[e]ments section.
Agreed. I think this helped during the "pitch period" as a differentiator for this proposal vs others. However, now its an adopted document the lone use of the term is odd. I'll prepare a PR that gives a nod back to the tus v1 spec that inspired this.
+1
I opened a PR for this: URL NAME Let me know if you had something different in mind.
This is a proposal to address issue /cc NAME
I've updated the graph in section 2 to note the new range. PTAL.
Each record currently includes between 1 and 256 octets of padding, while the default record size is set to 4096 octets. This is in many cases not sufficient, for example in the pad-to-next-power-of-two case. Extending the padding length to two octets will allow up to 65537 octets (65KB) of padding. I raised this on the ietf-http-wg list here: URL Martin about an upgrade path for existing users. I'll reply on the mailing list to not split the discussion.
This was fixed in and .
I guess that's the best we can do, considering that Unicode is a minefield. :-/
... in a swamp.
Is it worth dealing with the fact that 3629 doesn't actually define how to encode a sequence of characters, but we're calling it with one? We'd need to add something to this text, like "for each character (or code point?) in inputsequence, append the result of applying UTF-8 encoding to character to bytearray".
currently does not consider what happens when UTF8 serialization fails, although this could happen with surrogate points. I believe it should explicitly say that. Alternatively, the introduction of display strings could say that only unicode scalars (URL) are allowed (but even then I'd add the extra precaution in the serializer). Finally, we may want to add an example because all of this is far from obvious.
NAME and NAME pointed out that URL doesn't define the and operations that this draft uses. RFC 3629 only defines a character at a time. RFC 3629 also doesn't say to fail encoding or decoding in any particular situations. It says "The definition of UTF-8 prohibits encoding character numbers between U+D800 and U+DFFF", but its definition of encoding doesn't actually prohibit those characters. It also says "MUST protect against decoding invalid sequences", but the definition of decoding doesn't fail in that case. URL is more explicit on these questions and might therefore be a better reference.
Note that the recommended entry points for other specs are URL which are clear about how to handle BOMs as well.
Seems reasonable - will try to work up a PR. I'm likely to keep the general reference to 3629 for UTF-8, and use encoding for the specifics.
I would very very very much prefer not to have a normative reference on a "living" document here. Let's just add the missing needed on top of RFC 3629 to this spec. (and yes, I can make a concrete proposal)
We were clear how to handle BOMs, but what you suggest is a change on top of that.
See minimal PR.
(Posted on the PR by accident:) Is it worth dealing with the fact that 3629 doesn't actually define how to encode a sequence of characters, but we're calling it with one?
Refer to state-begin-advertising to describe the states when ULA site prefix is advertised.
has currently: The "as needed" part here is not so clear and needs to be detailed. E.g. only if an on-link prefix on the AIL is not present, the stub router provides one from its own ULA prefix only if the stub network does not have an on-link prefix or on-mesh prefix (whatever applies), then it creates one from its own ULA prefix. sizes of the involved prefixes? Better to either specify sizes or say it's up to implementation. state some randomness requirements - although for the ULA Global ID it may be clear that it must be randomized; we also have Subnet ID to have requirements for (or not). In any case state these assumptions.
This can be resolved easily by a pointer to other parts of the draft ("as defined in Section N"). Because there are still some pending PRs that impact the text here and add the details, this issue can be postponed until after these PRs are merged.