jiebi/CodeConvo huggingface dataset:
Created a separate issue to discuss ECN.
The guideline section discusses "Investigating a Range of Environments", with a focus especially on "Random Early Detection (RED) ] and Drop-Tail." The IETF and the IRTF have produced several different proposals for Active Queue management, with somewhat different characteristics. Interestingly, there is a tension there between developing Queue Management that "plays well" with existing congestion control algorithms, and developing congestion control algorithms that "take advantage" of AQM. Not sure what to say there, but at a minimum we should develop expectations about "signals" produced by AQM (e.g., the ECN markings of L4S) and reaction by congestion control algorithms. To go further, the could be a discussion of "honest signals", such as delays, that are very hard to "fake" in routing nodes, other signals such as ECN marks that could be produced by the network at very little cost, and packet losses that fall kind of in between. But we are not writing for the IRTF, so this could be kept brief.
On the topic of AQM, IETF published Characterization Guidelines (RFC7928) and Recommendations (RFC7567) that could be exploited for further discussions in this work.
Martin and Gorry: add a reference to 7567 to AQM behavior and evaluation criteria, which is not in scope for this document talk about evaluating CC interaction with AQM
Something like "CCs should be evaluated against AQMs that are prevalent in the internet" -- is there a reference that provides a taxonomy?
Editors: FQ-CoDel, PIE, L4S (RFC9332) have RFCs Are the others that should be evaluated, and do they have available specs?
Discussed at IETF 118: Many operators are not particularly aware of AQM or these concepts in general Include FIFO in the list Also do at least an AQM "you should evaluate the scenario with FIFO queues and should also consider the impact of various common AQMs and run detailed simulations against them", while maybe name-checking a few particularly relevant ones. Yes, should have criteria for this, and also should not ask people to run NxN interop matrix. Out of scope to mandate AQM
I suggest: Probably ought to include a suitably configured RED also.
and .
Also
The text currently reads: "The community should also consider other non-standard congestion controls known to be widely deployed," This is a compromise to land a mostly unrelated PR. NAME doesn't think it's fair to expect participants to test algorithms that aren't fully specified. I would argue that if such specifications exist and the algorithm is systemically important, it absolutely should be modeled. Moreover, if the algorithm is rather loosely described, in many cases there is open source code (often in a Linux fork!) that can be tested on a host, or in simulation frameworks like ns3. If an algorithm is a true black box, with no description besides marketing material, and no open-source code, I agree there is not much to be done. I also believe that while algorithms like this exist, they are not systemically important to the internet.
gorry: ns3 code is not super high fidelity.
Discussed at IETF 118: There's a danger if you see a realtime flow that goes faster than a best effort flow, since realtime is usually constrained by other factors (potentially needing time to move forwards to generate more data) Seems hard to imagine that something not specified is able to place a burden on anybody else. If you want others to do a thing, write it down and document it. Of course, there is no protocol police: we cannot prevent people from deploying whatever they want. Can we provide a carrot for people to play by the rules: Could be a SHOULD consider requirements and how to interact We have: Guidance for people who are writing specs that aren't yet part of the community How do we evaluate those protocols How do we ensure that fairness can exist with these Need to have a commitment from this WG to help and not just throw up new requirements for data and for those bringing new CCs to engage positively. Perhaps spelling this out clearly NAME to file issue to write something encouraging for people early on in the document Christian notes that separate queues between realtime and best effort are not actually a strong requirement To measure capacity you do need some sort of queue, or else you cannot measure
This info RFC has some sensible guidelines on RTP feedback frequency, if we need this: "Sending RTP Control Protocol (RTCP) Feedback for Congestion Control in Interactive Multimedia Conferences".
Related to
Condition (1) is concerned with the behavior with respect to Reno and, thanks to our edits, Cubic.
But there are of course other congestion controls out there, both standard and non-standard, in support of real-time and other non-greedy applications. Should we mention those as a class? Namecheck some examples?
editors meeting 8836 requirements document 8867 and 8868 are test cases, but informational. Reese: standard stuff is not widely deployed, widely deployed stuff is not standardized. But RMCAT is new. Maybe there aren't well-documented protocols to evaluate vs., so it's hard to require anything here. I'll take a look, will probably have to go to the community.
Editors: We need to consider the interaction between real time and not-real time is evaluating real-time in scope? See 8836, 8867, 8868.
Looks good enough.Thanks for the changes.
Comment by NAME I would suggest being explicit about the meaning of "active" and especially "deprecated": these are terms everywhere in RFCs (I am aware), but sometimes they are open for interpretation, and we keep discussing their meaning. It would be good to avoid that situation from this doc.
Comment by NAME The new registries do not have any guidance for the designated experts as to what registrations should be accepted or denied (conformance with template, point squatting, availability of reference specification, etc). See of RFC8126: The required documentation and review criteria, giving clear guidance to the designated expert, should be provided when defining the registry. It is particularly important to lay out what should be considered when performing an evaluation and reasons for rejecting a request.
In URL;list=ietf-http-wg, it was proposed that the PRI method be reserved in the HTTP/1.1 method registration document to avoid possible future reservation and ensure that the magic string remains a reliable way of differentiating HTTP/1.1 connections from HTTP/2.0 connections. There was a +1 and some offline discussion along the same vein, no dissent that I heard or saw. However, I don’t see it in URL, presumably because no RFC exists to reference. The appropriate doc is probably HTTP/2.0 rather than breaking the glass on the prepopulated registrations.
There was a question whether "SM" should also be reserved; I personally don't think so, since that line will never be interpreted as a potentially-valid HTTP request line or header. However, I don't think it would break anything if we did.
Break things, no. Squat on prime name-real-estate, a little.
Closes URL A client can express its interest in digests using during upload creation. The server can then respond with after POST and PATCH requests to share its digest with the client. The client computes its digest as well during the upload and compares the digest to spot issues. The representation digest of an upload resource covers the entire bytes that have been uploaded so far (i.e. right until the upload offset). NAME NAME Is this in line with the semantics of RFC 9530?
NAME Need a week for the review :)
Preliminary feedback: I'd simplify the language/split text on multiple lines; Describe processing in separate subsections, e.g.
Thank you for the feedback, Roberto. I added subsections for representation and content digests in URL I think this division helps the reader and also makes sense since representation digests can cover the entire upload, while content digests only cover one request. Let me know if you think differently.
URL added some guidelines on how a client may include integrity fields to upload requests for protecting the integrity of a single request or the entire upload. However, this either requires the client to compute the digest upfront (before sending a request or starting the upload), or the use of request trailers, which is not very widespread and not always possible (e.g. the ). Instead of requiring the client to compute the digest upfront, the server and client could calculate the digests while the upload is ongoing. Once the upload is completed, the server can send its digest to the client in the response to the request that completed the upload. The client can then compare the server's digest to its computed value. If they don't match, it can raise an error to the end user and decide to not continue using the uploaded data. The client should indicate its interest in the server's digest when an upload is created, so the server is not required to compute the digest if the client has no interest. I am wondering if the from can be used for this. Can a client include in the first upload creation request, but the server delivers the digest potentially later in a response for a subsequent request? An upload may be spread across multiple requests.
Should also be supported? For multi request uploads, there's really no use to continue if request 3 of 100 got corrupted.
As NAME mentioned in URL, is not directly applicable here because it asks the server to send the digest corresponding to the response content, not the received request content. Good point that a client may validate digests while a multi-request upload is progressing. A server could respond with the latest digest of the entire uploaded data (not just the data from the last request) using after POST and PATCH requests. The client can then validate and decide to abort if the digests don't match. This usage would be in line with the PATCH example in . The server responds with the digest of the representation after the PATCH has been applied. The example also notes that the server is not required to respond with the representation itself and could also have returned a 204 (as is done for resumable uploads):
Makes sense, thanks for clarifying. is most likely good enough given that it indicates the current checksum of what the server received.
NAME please, stalk me for a review in the next week :)
NAME It took me a bit more than a week, but here we go: URL Happy to hear your feedback :)
this is the part of URL related to -semantics
Continueing the discussion from URL Looking at the next text in context: To me it's unclear why we are calling out bare CR here, isn't the situation the same for LF and CRLF?? This also is not a new requirement, the ABNF does not allow CR and friends in field-value, and in 3.2 we say "A sender MUST NOT generate protocol elements that do not match the grammar defined by the corresponding ABNF rules." The subsequent paragraph talks about whitespace, and it may not be clear to everyone which characters we are referring to here (not the ones listed in Section 1.2.1 "Whitespace", correct?)
Proposal:
NAME ?
(forked from )
A bit iffy given the sentence above it, but okay.
This makes it clearer what standard is being depended on, while still deep-linking to the right section of that standard. This generates citations like "see of ]". The "RequestDestination" part is free text in the attribute. I don't see a way to change the "Part" text. We can use the to rearrange it to "see ], )" if y'all prefer.
Great, thanks for the PR. It's a lot cleaner than a bunch of independent references.
NAME just filed, thanks. URL URL
NAME have you asked the WHATWG folks for stable anchors to those sections? See URL for details of how to do that.
I haven't adjusted for vs. until that war is over. Note that I've explicitly added a format for the SETTINGS frame itself, which was missing.
+1 . This would make it easier to write extensions for both h2 and h3. Priorities draft suffers from the mixed presentation format right now.
OK, thanks for doing this NAME I merged this manually after fixing a few things up. From my commit: Use all caps for frame names Add a value for Type in frame definitions Change order of flag descriptions to match order in layout Make Exclusive field of PRIORITY optional Move PING ACK Flag to the right place Add SETTINGS ACK Flag Double-checking FTW. Of course, there's a good chance I also messed something up here. So feel free to check my work also.
I think the language about ordering is a bit weird, but it is clarified somewhat by the hex representations shortly after. I think those hex representations are doing the heavy lifting, as I’m not aware of any issue with interoperability caused by getting the flag indices wrong in practice. Nonetheless, if we’re really worried about it we could just delete “bit n” and replace it with “the x bit”, e.g. delete “bit 0” and replace it with “the bit”.
Switched from using the ABNF field types to the human names (i.e. sf-dictionary to Dictionary) for all of the Structured Field header values. Switched dictionary "type" to a Token (from String).
NAME could you PTAL at the change in referencing the header field types? Specifically, I wasn't sure if I should capitalize the types or not (I did since that's how the structured fields RFC referred to them).
Yep, this looks good. It's common to add something to you Notational Conventions section (or whatever) that says roughly "This specification uses the following terms from [ref to SF]: String, Byte Sequence, Token, ..."
This changes the maximum ZStandard compression window to be the larger of 8 MB and the size of the dictionary. This allows for clients to limit the maximum window size to as low as 8 MB if they like (by not advertising support) while also allowing for clients to use larger dictionaries with ZStandard than they would be able to if the global limit was 8 MB.
NAME and NAME could you also take a look and see if the language looks ok?
Context: URL aims to clarify the window size limit for the Content Encoding token as it wasn't clear in RFC8878. Since the Compression Dictionary Transport draft hasn't been published yet, we should also add some window size considerations to it for and to maintain interoperability. I think in Chrome we ended up using . NAME
NAME could you comment on how this impacts Brotli dictionary decode? Is the dictionary independent from the compression window (so we can adopt the same defaults as or do we need to specify something relative to the size of the dictionary? NAME NAME for ZStandard, to allow for delta-encoding large resources, is reasonable or should it be ? We have use cases for delta-encoding wasm applications where the dictionary can easily be upwards of 50 MB.
For Zstandard, the states that the entirety of the dictionary is accessible as long as the input is <= Window Size. Therefore, to give an extreme case, it's possible to have a 50 MB dictionary, and it's entirely accessible even if the Window Size is only 1 KB, but only as long as the data to de/compress is <= 1 KB. As soon as we reach 1 KB + 1, reaching the dictionary is no longer valid. For delta compression, this has consequences. Presuming the reference is ~50 MB, and the updated content is also ~50 MB, one would need a window size of ~50 MB to ensure access into the dictionary during the entire reconstruction process. But if the updated content is just a bit bigger, say ~51 MB, a ~50 MB window size would allow delta operation only during the first ~50 MB, the last ~1 MB would have to be compressed without dictionary. If follows that the size of the dictionary is irrelevant for the Window size. What matters is for how long one wants to access the dictionary while rebuilding the content.
And as a general approximation, one wants this dictionary during the entire reconstruction process, so the window size is effectively the size of the content to rebuild. Zstandard has a "windowless" mode which states exactly that : the size of the "window" is the size of the content. For this to work, one has to set the size of the content at the beginning of the compression process . Setting the content size is either an automatic operation when data is provided entirely upfront (), or, when data is provided in small increments (), it must be stated explicitly at the beginning (). will automatically trigger the "windowless" mode when the declared content size is smaller than the set Window size (effectively reducing the memory requirement for the decoder side). I'm not sure if there is another way to trigger it. Note that the decoder will be informed of the Window Size requirement by reading the frame header, and may decide to not proceed if this value is deemed too large. For example, has a default 128 MB limit, and will refuse to decompress payload requiring more that that, unless explicitly authorized to employ more memory (). Just for reference, offers a "large delta compression" mode with the command . It has made us somewhat familiar with some of these issues. In situations where the amount of data to compress and compare is very large, we now prefer to both the dictionary content and the destination file. It has proven useful to reduce memory pressure.
Thanks. That makes things a bit more complicated for the case where we want to require a minimum window size that the server can trust that all clients advertising content-encoding will be able to decode. For the dictionary case it might be cleanest to use the same 128MB limit for window sizes as the default zstd limit (knowing that the client won't allocate that much unless it actually gets a stream that uses it).
Hello. Shared dictionary is placed in the address space between backward reference window and static dictionary. In other words it does not interfere with normal compression window, but using it makes "static dictionary" references more expensive. We decided to order it that way, because shared dictionary works almost the same way as extra window. Sorry for the late response. Best regards, Eugene.
Extra note. As you can see brotli allows backward distances behind window.
IIRC, regular (as opposed to large-window) brotli allows distances up to ~56MiB, i.e. larger shared dictionaries would not be useful (== referenced)
Thanks. For the dictionaries to be usable for a broad set of apps (including large wasm bundles), I'm considering spec'ing: Brotli: Regular 16MB window, allowing for delta-compression of resources up to ~50MB. ZStandard: 128MB window, allowing for delta compression of resources up to around that size, depending on where the duplicated bytes are in the updated resource. The main difference with regular content-encoding is on the ZStandard side where the window is 8MB. For platforms that are resource constrained and are not willing to support a 128MB window they would be limited to Brotli (and then decide how large of a dictionary they are willing to keep and use). Another option on the ZStandard side would be to allow for a maximum window that is a multiple of the dictionary size (i.e. 8MB or 2 * dictionary size whichever is larger with a 128MB max). That way the client can have an upper limit that it knows in advance before advertising as being available. That adds complexity to the encoding side though. We have already seen cases where people want to delta-compress resources > 50MB (wasm apps) where they had to use ZStandard instead of Brotli because of the Brotli limits so I don't think a small limit like 8MB for ZStandard would work with this use case.
That massive resources exist is not sufficient justification to build a system that will not be available to some users. I would prefer a far smaller default. If only to encourage the use of smaller resources overall. Defining a separate codepoint for massive windows is possible, say . The scalable option also seems reasonable. Clients can then choose to offer dictionaries when they exceed available resources.
I'm ok using the same 8MB window ( I believe) that we use for for and note that the effectiveness of delta compression for responses > 8MB using may be limited. The scalable option concerns me that people are likely to get it wrong and it won't be obvious and the decompression will start to fail as the dictionary and resource size changes. NAME does that sound about right?
The alternative floated where 's window size is derived from the size of the dictionary is much more attractive to me. (That is, cap the window size log at , where the spec picks an somewhere in the range .) In such a scheme, where the window size is deterministically derived from the size of the dictionary--a resource the client must already have access to--the client is free to decline to advertise support for for dictionaries which would require the use of a window larger than the client is willing to support. I.e., in resource-constrained environments, Chrome could avoid having to allocate more than 8 MB windows for by not advertising support for on requests where the resolved available dictionary is larger than . Put another way, given that the client will have the tools to make these choices for themselves, I would rather give the client discretion here rather than just bake a flat restriction into the spec.
NAME would you be supportive of updating the zstd command-line tooling to add an option that would set the window to the larger of 8MB or the dictionary size (i.e. match the content-encoding behavior)? My main concern is that it is easy to document and get people to use a fixed command-line for a 8MB window but if it involves a more complicated algorithm then it is likely to break in subtle ways that they may not notice until the resources are bigger than the dictionary (and are bigger than 8MB).
NAME yeah, that would be pretty straightforward. Happy to add that.
If we end up that bundles the dictionary hash with the compressed stream we can make sure the tooling for that file format handles the window size stuff for zstandard automatically (my main concern is around unexpected breakage from different server and client window requirements that only trigger when resources grow). The language will be a bit complex but will probably have to look something like
If I may chime in for some tactical consideration : In an update scenario, it's common for a new resource to be slightly larger than the old one it replaces. The expectation is that we generally tend to add stuff over time. A formulation which caps the window size to the size of the dictionary would introduce problems for all these update cases, which are expected to be the majority of update scenarios. On the other hand, we don't want this increase in size to be completely uncapped, in order to keep control of memory size.