jiebi/CodeConvo huggingface dataset:
Note - this is a rough draft, although I have done some edits while adding this material. I do have notes for to-dos, and I'm pretty sure I want to figure out how to do the diagrams in some way that will format with straight lines, for starters.
I'm happy to just merge this in now as is just to unblock other work and make progress, and make any adjustments later because there's nothing that stands out as controversial, NAME are you okay with this?
We have a plan on how to resolve our terminology migration for in this PR
Based on review comments from NAME on the mailing list .
Draft is in suhasHere/moq-drafts.
NAME - NAME and I are starting with the scenarios from URL
NAME - I think it would be good to close this issue when we have a reasonable plan for where the material from the scenarios draft goes, and then handle further changes via issues and PRs as usual. Does that make sense? NAME NAME and NAME - I wanted to let you know that NAME and I are (finally, finally!) merging text from the scenarios draft in the requirements draft, as NAME suggested. I think I have homes for almost all of the text from the requirements draft, but I do have some work to do, so that the material is in the right place, and that it flows with text from the previous version of the requirements draft. I am, of course, always interested in comments. And thank you each for the work you did on this material.
Awesome - let me know if something I can do to help
Thanks NAME .. look forward to the updates. +1 to NAME , please let me know if we can be of any help Here is my review. Firstly, I'd like to say that this document is badly needed. The requirements document doesn't get into the level of detail that this document does, and I think that this level of detail is important to make progress on the design. Terminology I think it would be useful to define some of the terminology, including terms like "track" and "media switch". Section 2.2 A few questions: Are the trackIDs "t1, t2, t3" separate MediaStreamTracks, or are they just multiple encodings of the same track?
Are there scenarios in which it would be useful to be able to ingest multiple MediaStreamTracks (either audio or video)? I ask because the RUSH draft doesn't support ingestion of multiple tracks and I've been told that this is a problem by some customers. Section 2.4 A media switch, sourcing tracks that represent a subset of tracks from across all the emitters. Such subset may represent tracks representing top 5 speakers at higher qualities and lot of other tracks for rest of the emitters at lower qualities. [BA] The use of the term "media switch" suggests to me that it is forwarding tracks, rather than sourcing (e.g. transcoding) them. Can you clarify? subscribing to subset of the tracks [BA] I think there may be an implicit assumption here, which is that the receiver only indicates the track it wishes to subscribe to, rather than indicating the quality it wishes to receive. Above setup brings in following properties on the data model for the transport protocol [BA] The paragraphs that follow appear to describe different ways that the system could be implemented, rather than "properties on the data model". Do you mean to say that the design should be able to handle any of the approaches described? Media Switches to source new tracks but retain media payload from the original emitters. This implies publishing new Track IDs sourced from the SFU, with object payload unchanged from the original emitters. [BA] Is the term "source new tracks" implied by changing the Track IDs? If the payload is retained, this seems like forwarding to me, rather than "sourcing new tracks". Media Switches to propogate subset of tracks as-is from the emitters to the subscribers. This implies Track IDs to be unchanged between the emitters and the receivers. [BA] What does "as-is" mean here? Are you referring only the lack of a change in the "Track ID"? If the switch drops some frames in order to reduce quality (and bandwidth consumption) is it still "as-is"? Subscribers to explictily request multiple appropriate qualities and dynamically move between the qualtiies during the course of the session [BA] Is it a goal to be able to support such a "receiver driven" approach, as well as the other two approaches, which are sender driven?
Section 3.1 For example, rather than waiting for 30 seconds before connecting, the user might quickly download the "key" frames of the past 30 seconds and replay them in order to "synchronize" the video decoder. [BA] I think this sentence points out that the term "access point" may need to be clarified, because in such a scenario, the term "access point" may have a different meaning in stored media than in media transmitted over the wire to speed up join latency. Section 3.2 It is possible to use groups as units of congestion control. When the sending strategy is understoud, the objects in the group can be assigned sequence numbers and drop priorities that capture the encoding dependencies, such that: an object can only have dependencies with other objects in the same group, an object can only have dependencies with other objects with lower sequence numbers, an object can only have dependencies with other objects with lower or equal drop priorities. [BA] This text seems to be mixing distinct concepts such as dependencies and layering. In layered coding, there is typically only a requirement not to depend on higher layers, but it's possible to depend on frames in lower layers. Mixing drop priority (a transport concept) with groups seems confusing to me. The main drawback is that if a packet with a given drop priority is actually dropped, all objects with higher sequence numbers and higher or equal drop priorities in the same group must be dropped. [BA] Leaving gaps in the sequence number space does seem problematic, but that's a choice, not a requirement. If the group duration is long, this means that the quality of experience may be lowered for a long time after a brief congestion. If the group duration is short, this can produce a jarring effect in which the quality of experience drops perdiodically at the tail of the group. [BA] What does "group duration" mean? Are you referring to time between I-frames? This paragraph seems to assume that group duration is a constant rather than something which can be set dynamically. Section 4 Some video codecs have a complex structure. [BA] Are you referring to codecs, or encodings? It would send for example: [BA] A picture might help here.
I think you're referring to the scalabilty mode known as "L2T2h": URL Of course, we could encode these dependencies as properties of the object being sent, stating for example that "object 17 can only be decoded if objects 16, 11 and 7 are available." However, this approach leads to a lot of complexity in relays. We believe that a linear approach is preferable, using attributes of objects like delivery order or priorities. [BA] Why would this lead to complexity in relays in a receiver-driven scenario? The relay would only forward the layers that the receiver asked for, and wouldn't have to store state relating to the receivers. Also, the real issue here isn't dropping layers, which can occur at any point. The reason dependencies (and more) are needed (and why the "linear approach" doesn't work) comes when it is desired to add layers. It is of course possible just to start forwarding additional layers, but that doesn't improve quality unless the forwarded frames can actually be decoded. To figure this out, whoever is in charge (receiver or relay) needs to be able to determine the "upswitch points". This also implies feedback from the receivers to the encoder (e.g. LRR). Section 4.2 We propose to express dependencies using a combination of object number and object priority. [BA] How would this permit restoration of video quality after congestion clears up? I don't think you have enough information. Section 4.3 . Relay behavior In case of congestion, the relay will use the priorities to selectively drop the "least important" objects: [BA] Dropping is easy, it's adding layers back (and decoding them) that's hard.
Frames can be coalesced or split -- this is antagonistic to end-to-end, per-frame compression. This requires decompression / recompression in proxies that need to coalesce or split frame.
Discussed in NYC; no implementer interest, security concerns about proxying it. We believe it can be addressed as an extension now if someone wants to scratch that itch (and can find someone to interop with); however, fixing this properly means doing it at the HTTP semantic layer (e.g., with a new range type). In the meantime, remove frame compression.
to use the new "validator" name. See: URL URL
NAME I'm quite weak on that side: is this PR correct? If not feel free to fix at will :)
NAME I agree... this comes from the original RFC 3230. Merging the change as a simple terminology update. The relation b/w validators and the resource is discussed in , where I tagged you as your guidance will be great. See too.
to clarify the following passage for defining the behavior of digest with caching (eg. when Digest is returned together with Last-Modified and ETag). and to better extend the definition of Digest to POST and PATCH requests. This is because though only when they include explicit freshness information. If we go on the new http-core definition (which is cleaner) we have that eg. According to URL POST responses identify returned resources in different ways according to the following: the primary resource is identified by Content-Location the enclosed representation matches the primary resource eg. 1 eg. 2 primary resource identified by enclosed representation matches the status of the request according to the resource's own semantic eg.
re-reading the old RFC my understanding is that validator fields should just be aligned with Digest (that is: digest-value should be aligned with the values of last-modified and etag). While it is trivial, the original editor could have considered that a potential issue.
The change itself looks right. I wonder however about the whole paragraph: what does it mean to say "resource is specified"? Doesn't that also need request header fields upon which the returned representation varies?
try to say that for multiple post-handshake client certificate authentications, only use the last one (to
TLS 1.3 post-handshake authentication enables the use of multiple client authentications. The draft probably needs to address this. Firstly, the proxy could be smart enough to know which certificate applies. After all, it needs to know how and when to ask for a certificate. If it is that smart, we don't have a problem. If not, then... For , this is easy. If there are multiple certificates and the reverse proxy doesn't know which to include, it can include all of them. It's easy enough to add extra values. For this is less easy. There would be no way to distinguish which intermediate related to which end-entity certificate if you just added them all. I don't think that's fatal though. Chain building for validation always has to deal with certificates that don't do anything, so maybe we can just dump them all in there and let the back-end server sort them out. In terms of actions, we can say something like:
The draft currently , which precludes multiple values/instances. I am very reluctant to change that in support of something that rarely if ever happens in practice (it would mean HTTP/1.1 with multiple TLS 1.3 post-handshake authentications or post-handshake authentication(s) after a client cert authenticated handshake). Could we instead just say the first/last one is used? Then it's predictable/defined should it ever happen. Or declare it out of scope. Existing related text from URL is copied here: >HTTP/2 restricts TLS 1.2 renegotiation ( of ]) and prohibits TLS 1.3 post-handshake authentication ]. However, they are sometimes used to implement reactive client certificate authentication in HTTP/1.1 ] where the server decides whether to request a client certificate based on the HTTP request. HTTP application data sent on such a connection after receipt and verification of the client certificate is also mutually-authenticated and thus suitable for the mechanisms described in this document.
I could live with picking the last in the absence of information that might allow for another to be selected. But what harm is there in using a list?
Makes it all more complicated b/c everything then needs to treat it as a list and reason about it being a list.
Is this PR meant to include ?
Yeah, oops. Need more coffee.
Try now.
Otherwise, looks alright to me.
The current definition of sh-float is surprisingly hard to serialize correctly, the best C-code I could come up with is: int serializeshfloat(char buf[18], double d) { int p = 0, l, dig; buf[0] = ''; buf[1] = '\0'; if (isnan(d)) return (-1); if (fabs(d) >= 99999999999999.95) return (-1); if (d = 99999999999999.95) return (-1); if (d < 0) { buf[p++] = '-'; d = -d; } p += snprintf(buf + p, 18 - p, "%-16f", d); while (buf[p-1] == ' ') p -= 1; buf[p] = '\0'; return (p); }
+1 from me, our primary focus should be on data types that will have good interop. That level of precision isn't necessary for any current use of HTTP headers I'm aware of, off the top of my head.
Proposal: Call these decimal. Allow a maximum of 9 digits on the left of the decimal place and 6 after. This matches the definition for int and allows storing this in the same sized slot with a or conversion factor.
ABNF gets much simpler.
NAME That would probably curtail the range too much. I think it is desirable to be able to move posix time_t timestamps in these.
If this really is an integer with fixed point semantics, why do we need it at all?
Because it's easier than making people specify "divide the number by a million to get its real value"? That said, I agree that 9 digits to the left of the point is probably not enough, so a floating point representation is more useful. This is the ABNF equivalent of NAME C code, yeah?: That easily gets us a current 32-bit time_t plus microseconds, and a whole range of other values besides.
Why do the arguments about size of integers not also apply here? is 64-bit on my system, which doesn't fit in the sh-integer definition either.
The current time is ish. You don't need to prefix it with 30 bits worth of zeroes.
I've read the IETF105 minutes, and thought about the fixed-point discussion. All the uses for "decimal number with fractional part" I can think of\ would reasonably served by at least 10 digits to the left\ and at least 3 to the right of the point. So I would support any fixed- or floating-point representation that supports that range of values. \ that's all that matters, right? \ a ten-decimal-digit time_t gives us until November 20, 2286
Using the max-six-decimal definition, and allowing 15 digits with no period or decimals, this C-code will do the job: static char * sh_float(double d) { static char buf[40]; int n; if (fabs(d) < 1000000000) { n = snprintf(buf, sizeof(buf), "%f", d); while (buf[--n] == '0') buf[n] = '\0'; if (buf[n] == '.') buf[n] = '\0'; } else { n = snprintf(buf, sizeof(buf), "%.15g", d); } return (buf); }
Sorry, wrong button.
If I take , then I can almost represent that in a JS float in the most horrific way imaginable: But that doesn't quite fit. 69999999999999999 doesn't fit into a double. If you want to remember the number of additional decimal places so that you don't lose any precision on those trailing decimal places (things like 0.3 are surprisingly tricky to re-encode), you are left with supplementary values. Can someone make a better case for any more than 10^10 values here? I couldn't see a good one. You can fit a 32-bit value in that. If you really think that time_t is valuable, which I'm totally not convinced of, then fewer steps might be acceptable. I can pack this:
PR updated. Any further comments, or ready to merge?
The CONNECT-TCP draft makes informative references to other drafts which use "extended CONNECT" (CONNECT-UDP, CONNECT-IP), but does not have a normative reference to the draft which defines this use of the CONNECT method. It references the "extended CONNECT" method without supplying a reference for this term. This draft spells out the required values for each pseudo-header, but omits the requirement to set SETTINGSENABLECONNECT_PROTOCOL. A reference to RFCs 8441/9220 is probably needed, and a reminder that the method in this document can only be used if the server sends this setting first.
That'll do.
by copying the text from DATA
says: I think the intended reading is that when the parser reaches the point that only the FBF and the Padding remain, the length of Padding is given by Pad Length and the fragment is the balance. If Pad Length exceeds the remaining length of the frame, it's an error. There is a less-likely reading also allowed by the current phrasing, that Pad Length can't be more than half the remaining bytes (else Padding would exceed the size of the field block fragment). That's very unlikely to be the intended reading, but we should be clear here. I'd suggest simply taking the note from the end of DATA and putting it here verbatim.
LGTM, will let NAME take a read as well.
NAME I could also include some other non-requirements into the section if you have some to mind.
A similar section seemed helpful during discussion of URL My first nominee (and I'll put that in the corresponding PR) is that we're assuming that we're working with existing media transport protocols like RTP and SRP, rather than inventing a new media transport protocol in the IETF, but we can talk when I have text.
We have this in place, and I think we have enough non-requirements to start with, let's keep adding as appropriate.
This PR moves track request params to its own section. This was messed up during huge PR merge as part of preparing for IETF117 work. Also it add track namespace length field to Announce_OK message which was omitted.
NAME good catch. updated the commit to address your comment. Thanks
It's an improvement so we should merge, but we do need to clean up how parameters are encoded.You may also want to move the actual definition of Track Request Parameter, it's currently in the SUBSCRIBE REQUEST section.
I do not agree with this suggestion. Upgrade is intended to support a not-yet-defined future. Limits placed upon it must be evaluated in terms of what will be excluded from future use, not what is currently being used in practice. If you want to limit what can be sent in a body, do that instead. The entire premise of this document is that a client will send something incredibly stupid and uncontrolled in a request body while asking for an upgrade. Just forbid that and be done. In any case, Upgrade with GET, HEAD, and OPTIONS has no such issues. Upgrade should also be possible to use with QUERY with any compliant server. If you can find a server that fails to process a valid request body and then proceeds to treat it as a pipelined request, that is a vulnerability in their code.
This PR originally assumed the deprecation of "TLS" in . I've removed that deprecation there, so other HTTP methods are necessarily allowed. Due to that change, I've rewritten the recommendation here, removed the PR stacking, and removed the "draft PR" flag.
Formally, Upgrade can be attached to any HTTP request, in which case the request is processed in HTTP/1.1, and the server has the option to Upgrade before returning the response. This includes requests with and without nonempty bodies. Handling of Upgrade with a nonempty body is well-defined, but the Upgrade paths that are actually implemented today (WebSockets and MASQUE) only use GET requests with empty bodies. NAME raised concerns that this is something of a "sharp edge" for security. For example, a server implementation might assume that there is no body in an Upgrade request, leading to a mismatch between the client and server when a body is present. We could make this problem go away by mandating that Upgrade only be used with "Content-Length: 0", and perhaps only with "GET". However, this requirement would appear to contradict RFC 2817, which says "A client MAY offer to switch to secured operation during any clear HTTP request", so that document would have to be Updated or marked Historic.
FWIW HTTP/2 upgrade has some statements that might conflict with the proposal. From URL before the client can send HTTP/2 frames. This means that a large request can block the use of the connection until it is completely sent.
important, an OPTIONS request can be used to perform the upgrade to HTTP/2, at the cost of an additional round trip. On the other hand, RFC 9113 , so the topic of this issue is compatible with that notion with a bit of wordsmithing.
Well phrased. More concretely, for servers which read (and buffer) the full request -- including request header and request body -- before processing the request, this is a non-issue. However, many servers make routing and other decisions after the request headers are received. To handle upgrade after the request has been handled, all requests would need to check for Upgrade at the end of handling requests, not at the beginning, which is more common in code that I have seen and in code that I have written. I think it would be ideal if Upgrade requests for new Upgrade tokens were sent to endpoints for which an alternate resource is not provided if the Upgrade does not succeed. If a request with Upgrade is rejected, the HTTP status could reflect that. At the moment, a client can request "GET /altresource.mp4 HTTP/1.1" with "Upgrade: websocket" and the RFCs specify that /altresource.mp4 should be delivered in its entirety before the server switches to WebSockets, and the HTTP response headers included Upgrade: websocket, along with Content-Length or Transfer-Encoding: chunked for /altresource.mp4. The combination of request/response bodies for resources tangential to the Upgrade request is what I would like to see avoided in future Upgrade tokens. As NAME worded so well, I think there are sharp security edges here, and complications in implementations. HTTP/2 extended CONNECT is a better approach as it is only the :protocol change request, and does not include a request body or a request for an alternate resource, at least as I understand RFC 8441 to describe HTTP/2 extended CONNECT for . Another example of where request bodies can potentially be mishandled is if an origin server passes the Upgrade request to a backend via CGI, FastCGI, SCGI, reverse proxy, or something else. Even if the origin server handles the request body properly with the Upgrade request, I certainly do not trust an amateur PHP program to handle the request body properly before switching protocols.