jiebi/CodeConvo huggingface dataset:
I think this is the tip of a larger iceberg, and I'd suggest we'd need to go deeper, if the IETF now thinks they may produce specs in this space. As I recall (others may add to the list), there are challenges to be considered by a proponent/IETF, which include: 1. Does the proposal rely on technologies specified by other Forums/SDOs? 2. Does the IETF have sufficient experience of the DC requirements and to review the proposals? 3. Does the proposal include experience of interoperability between different implementations? or a plan for this? 4. Does the DC traffic share resources with Internet traffic? 5. Specifically: does the method to be used between DCs or only within DCs? - How is this usage scoped? And what happens when the protocol is bridged across an Internet path? One or more of these has in the past thwarted progress towards adoption in the IETF, can we say something about this?
Do you want to take a shot at it? Do we need a completely different section about DC algorithms?
OK, it might be good to keep some version of this as a guidance for evaluating general-purpose protocols in datacenters, AND have a totally different section for DC-specific algorithms.
From IETF 119: Data centers with really small RTTs should be discussed as a special case
NAME We need to be specific about what we mean by datacenter (e.g. Hadoop queries), please give some details about the data center scenarios that we want to take into account NAME Typically you see proposals that rely on the fabric doing non-internet-y things, often there's combined implementation with the data plane, so we need to be specific there. Also saw some presentations this week about non-ethernet fabrics, need to describe what you see from your fabric and what you can then do from a congestion control perspective. NAME Some spillover from IPPM, doesn't work on public internet and not intended to. Number of things being deployed today, massive clusters, 100,000 endpoints constantly talking to each other. Special congestion controllers are very much needed. Trying to keep CPU/GPU non-idle all the time. We need something that is fast within 1-2 RTTs, doesn't matter if done on endpoints, NICs, switches.
This makes it clearer that they can't be sent otherwise.
LGTM
Merged by RFC editor.
has the following clarification added in draft 17: Should this be instead of ? If not I think we have an inconsistency with which has the following for the section:
NAME - Ping.
This is merely a problem with perspective. As in, there is none. I can provide that thusly: Does that work?
This is an improvement. I'm not sure if we can avoid it but it is a bit confusing with respect to :
I don't think that's a problem. The (remote) peer isn't sending, but the (local) peer can.
Agreed. Ship it :) Thanks!
: As this sentence seemed to indicate that the sender itself was sent, we updated the text to clarify that the GOAWAY is sent. Please let us know if this update is incorrect. Original: Once sent, the sender will ignore frames sent on streams initiated by the receiver if the stream has an identifier higher than the included last stream identifier. Currently: Once the GOAWAY is sent, the sender will ignore frames sent on streams initiated by the receiver if the stream has an identifier higher than the included last stream identifier.
This change does two things: change the SHOULD to a MUST when specifying what size of prefix to request, so that only requests for /64 prefixes are allowed, and providing a reference to RFC8415 that says how to provide such a hint. Additional text describes what to do if there is more than one prefix.
OLD: A SNAC router SHOULD request stub network prefixes with length 64. New: A SNAC router should request stub network prefixes using IAPD with IAPrefix with a hint of 64 as defined in RFC 8415 Section 18.3.9.
Agree with adding the details. Is there a reason to change SHOULD -> should ? Seems wise to keep the SHOULD in caps given the MUST requirement that follows soon after it:
This editorial change attempts to per feedback from NAME
The three steps from Section 4.3 on Registration are essentially the same. To verify the signature you have to retrieve the issuer credentials and you have to do item 4. Issuer Verification: The Transparency Service MUST syntactically validate the Issuer's identity Claims, which may be different than the Client identity. Signature verification: The Transparency Service MUST verify the signature of the Signed Statement, as described in [RFC9360], using the signature algorithm and verification key of the Issuer. Signed Statement validation: The Transparency Service MUST check that the Signed Statement includes the required protected headers listed above. The Transparency Service MAY verify the Statement payload format, content and other optional properties.
LGTM for reduction and lack of duplicity
See issue
Closing issue
Add reference to VC presenation specs in section 4 and section 2.2.1
Only adding references to section 4. Adding this in section 2.2.1 will be confusing, even though a QR code is being scanned in these protocols, unlike device code flow, the session does not revert back to the consumption device.
Addressed in PR
Looks good to me, one small editorial change.
There's an assumption in the current draft that some messages are partially reliable (OBJECT messages) while some are fully reliable (everything else, aka control messages). The ability to drop a message depend on the ability to reset the QUIC stream. However, reseting a stream containing control messages wrecks havok on the subscription state machine. There's no way within QUIC determine if a messages was received or processed prior to a reset. For example, if I send a SUBSCRIBE on stream and then a receive a STOPSENDING for that stream, then the subscription enters a limbo state: The SUBSCRIBE was processed and a SUBSCRIBEOK or SUBSCRIBEERROR is forthcoming. The SUBSCRIBE was never processed and a SUBSCRIBEOK or SUBSCRIBEERROR will never be received. The only option currently is to set a timer after a reset is received. After a few RTTs or so, we assume the SUBSCRIBE was lost and issue a new SUBSCRIBE on a new stream, introducing latency in the process. Though things get extra weird if we receive a response after this timeout expires... I think we either need to state: Control streams MUST NOT be reset (like HTTP/3) Reseting a control stream also resets all prior messages (ie. all prior SUBSCRIBEs). Reseting a control stream resets all unacknowledged messages (ie. SUBSCRIBEs without SUBSCRIBEOK). Option 3 is problematic, since the subscriber knows that the OK was received prior to the reset but the publisher doesn't. Personally, I think option 1 is the best is there's a single control stream, while option 2 is the best if there's multiple (scoped) control streams. Regardless of the approach, data messages and control messages should be split () because they have different reliability. A receiver should know if it can issue STOP_SENDING for an OBJECT without impacting control messages, especially if we take approach 1. My application uses a bidirectional stream for control messages and unidirectional streams for data messages. It makes it very easy for the application to know if a stream is resettable just based on the ID.
From , (s/channels/streams) -- snip Control and Data Channels When a endpoint client or relay begins a transaction for media delivery, a new bilateral QUIC Stream is opened.
This stream will act as the "Control Channel" for the exchange of data, carrying a series of control messages in both directions. There is one "Control Channel" setup per track. Control Channels are "one way" and are setup hop-by-hop between the participating MoQ entities. If a peer both sends and receive media, there will be different control channels for sending and receiving. For exchanging media, OBJECT messages ({{message-object}}) belonging to a group are delivered over one or more unidirectional streams, called data channels, based on application's preferred mapping of the OBJECT and its group to underlying QUIC Stream(s). The control channel will remain open as long as the peers are still sending or receiving the media. If either peer closes the control stream, the other peer will close its end of the stream and discard the state associated with the media transfer. The latter resulting in termination or clean up of stream states associated with the data channels. -- snip The meta level thinking there was , OBJECT messages sent over data streams doesn't impact control stream. However closing a control stream, say for a track namespace, will end up closing all the data streams corresponding to the media transfer for the associated track namespace
Resetting the control stream should be an error, given there is only a single control stream.
The client creates a bidirectional stream and sends the SETUP message. The server replies with its own SETUP message on the same stream. Afterwards, both endpoints can create any number of unidirectional streams containing any non-SETUP messages. Messages with different priorities should not be mixed in the same stream, which means control messages should never be mixed with data messages. Messages can't be sent or have to be buffered until the receipt of the SETUP message. Control messages over separate streams arrive in an unknown order.The client creates a single bidirectional control stream. The control stream MUST start with a SETUP message, followed by any number of control messages (ex. SUBSCRIBE). The sender creates unidirectional data streams. The data streams MUST only contain OBJECT messages. Pros: Partially Control messages are no longer intermingled with with data messages, and the control stream MUST always be the highest priority. Endpoints no longer have to buffer messages, as QUIC will automatically make sure SETUP arrives before any other control messages. Data streams MUST NOT be created until receipt of SETUP.
Control messages arrive in a specified, deterministic order. Cons: Head-of-line blocking between control messages, which sometimes may not be desired.
While I was reading the requirements draft, I found some use cases mentioning bidirectionality. Is it expected/supported that there can be data flowing in both directions (eg: a p2p call)?
Yeah, my intent is that there's a "sender" and a "receiver" per track. The ROLE param in the SETUP message indicates if an endpoint (client/server) intends to send tracks, receive tracks, or both. In the bidirectional case, both endpoints would say ROLE=BOTH. Then they could independently PUBLISH or SUBSCRIBE to tracks. Alternatively, we could say that tracks may be bidirectional... but I don't think that makes any sense.
We can simply say that data objects MUST NOT be sent over control streams, and remove any reference to whether data streams should be unidirectional or bidirectional. It doesn't matter and we can leave it to the end-points discretion.
The proposal above aligns pretty well with the idea in , more specifically this I like the separation of control stream and data streams, the latter being unidirectional streams.
I like the proposal for this. I view the HOL blocking of control messages a feature not a bug. More specifically, I think that some of our messages need to be delivered in order. Specifically if a client is trying to stop and high res video subscription and start a low res subscription, it would be good to ensure that the subscribe to the low res track does not happen until after the end the end subscription to the high res track has happened. Sending the un-subscribe then subscribe on one quic stream allows these to be sent and same time to server ( we don't need a round trip between them ) but still be delivered in order. I think it needs to be clear that multiple control messages can be outstanding at once. For example can send multiple subscribes before getting the response to the first subscribe.
I would like to write up a PR based on the above ( basically resurrecting ) that pertains to the control channels.
I like the idea of seperation of control stream and data stream.
WG is ok with a single stream, with no critical use cases for multiple. A single stream makes it easier to change bitrates 'atomically'.
People don't want Object messages allowed on the same stream as control messages. People noted that this means the only bidi stream is used for the control stream in practice, and all unidirectional streams are used for objects, but people don't want to bake that into the spec.
The current draft specifies that control messages may be sent over separate streams. This means that control messages will arrive in an unknown order. This can create divergent behavior if messages depend on any form of ordering. The simplest example if sending a SUBSCRIBE message followed by a later UNSUBSCRIBE message on a separate stream. If the UNSUBSCRIBE arrives first for whatever reason (packet loss or poor prioritization), then it has a completely different meaning. The current draft adds some advisory text that control messages should be on the same stream if ordering is required. I think we should put all control messages on the same stream, so ordering is enforced and the behavior is deterministic.
I think life will be way easier if all control messages are reliable and in order. So +1 on single control stream. One of my arguments for this is the difficulty in even testing our of order control messages.
Will be fixed by the PR that
As mentioned in , there's currently no distinction between control streams and media streams. Should we have dedicated control streams? When should these be cached or forwarded by relays? For example, GOAWAY must NOT be forwarded by a relay, but something like TRACK (does not exist yet) should be forwarded.
This sounds like something worth thinking on and capturing more about in the spec.
QuicR supports control streams and media streams. Control Stream is for setup and other control features Media Streams for delivery media - today it supports QUIC Streams or QUIC Datagrams. More details can be seen here : URL This keeps protocol design cleanly separated and also extensible
Discussed today, there was interest in separating today's control messages from media/object streams, but not yet interest in constraining a MoQ session to a single control stream. Per track and per namespace were both mentioned as reasonable scopes, as well as 1 per session. There were follow-up questions about Setup messages and version negotiation and how you know which version is in use.
One conclusion from the chat is that QUIC streams MUST NOT contain messages with different priorities.
Otherwise, a QUIC library would need the ability to reprioritize a stream at a given byte offset. If control messages are always higher priority than data messages (debated), then separate streams are required. I also mentioned that in my implementation, separate threads process control and data messages separately. Data streams are immediately sent to a media decoder thread (WebWorker) while control streams stay in the main thread. This architecture is nice but breaks if control messages and data messages can be interleaved. I think we were in pretty good alignment that there should be control streams. The primary discussion surrounds the scope and number of control streams. control stream per track. control stream per track namespace. control stream per session. I don't think option 1 is viable for ABR, as you want to coalesce the UNSUBSCRIBE / SUBSCRIBE for separate tracks. For example, (near) simultaneously unsubscribe to 480p and subscribe to 720p. If the tracks are truly independent and can not tolerate head-of-line blocking, perhaps they should use a separate namespace instead? I think option 2 is acceptable. The goal is to avoid introducing head-of-line blocking for thousands of unrelated track namespaces, but I think this bleeds into the infamous pooling discussion. If pooling is accomplished via multiple sessions like WebTransport, then option 3 also accomplishes that goal. I'm currently using option 3 since it makes the SETUP marginally handshake easier. Control messages are written immediately after the SETUP message, avoiding the need to buffer messages pending the handshake. It also means you can coordinate subscriptions when tracks are NOT within the same track namespace (ex. ABR across different producers).
Okay so I can't help but start a tiny bit of the POOLING discussion. My application uses Warp for media and Alan's protocol for chat. Let's assume that I want to use the same QUIC connection for both for efficiency reasons. I would like to design my application into two halves, so one part of the code deals with chat and one part deals with media. I don't want a situation where I have to pass control and data messages between threads. It would be really nice if there were two control streams: one to control the chat protocol and one to control the media protocol.
This would also be nice for the CDN, as each control streams could be routed to a separate backend/origin. A fatal error (RESETSTREAM) on the chat control stream would not impact the media control stream and vice-versa. However, this doesn't quite work right now because the can be collisions. The chat and media system would need to coordinate to avoid collisions, otherwise IDs need to be rewritten. Any incoming data streams need to be forwarded to the correct component based on this rewrite lookup table. One way to fix this is to extend all messages to include a , where 0=media and 1=chat in this example. Each control stream gets its own ID and OBJECTs would include the ID, avoiding collisions and making it very easy to route within the application and CDN. (this is basically WebTransport session pooling) In fact, now each session could even use a different version of MoQ. This would be super useful when the chat and media backends are running different versions of the protocol. It's why I suggested at the end of the call (NAME that each control stream contains it's own SETUP handshake.
Individual comment: Do you mean the "sendOrder" field in OBJECT (section 6.2), or something else? I'm not sure normative language is required here, especially if it's based on what current QUIC implementations supports as a prioritization API. Someday a QUIC implementation could support changing priorities at different offsets for a stream. As for pooling - WebTransport has taught us that pooling is a huge headache. I suggest we defer to WebTransport or open multiple QUIC connections rather than trying to define yet another pooling layer in moqt.
I have a draft in the works to support doing this for HTTP/3, so it's not far off in my world
Well, I kinda think the normative language is justified when priority schemes are written on the wire. You're totally right that an implementation could prioritize at byte offsets... but that doesn't mean a relay can support it. Ugh, I guess we need to discuss if prioritization schemes are negotiated or not. My goal was to require a base scheme (send order) in the transport draft so we could just assume that every MoQ relay will support it, and thus normative language like this is fine.
But you're right, new prioritization schemes will absolutely come along and maybe send order should be a guinea pig. So maybe: if the is negotiated, then messages on the same stream MUST use the same value. I believe the same is true for NAME prioritization scheme. Either way, point taken and it was a bad argument on my part, and barely even related to the topic of control streams. Gross but point taken. I assume the priority can only decrease?
The design doesn't currently implement that constraint, the reason being that an application using HTTP might have a need to increase the priority of some bytes relative to other streams.
Going back to the topic of control streams, we need to support both the following Control stream per track namespace One Control stream for the session - (for setup, termination) Also i feel the transport draft should have the text for it as well to explain what happens when a control stream for a track namespace is created or reset and how are they related to the data streams carrying the objects.
NAME mentioned my priority scheme. On the prioritization scheme, my personal thinking evolved as I was studying implementations and consequences. For a while, I did believe that layered encodings would be better implemented by having a single track for the whole media, and having for each object priority fields of some kind indicating the relative urgency of the object. I think now that I was wrong, because it leads to a complex mapping that makes relaying harder, and also because the exact semantic of the "priority field" is too complex to specify in a base protocol. The transport mappings get much simpler if all objects in a track are treated with the same precedence -- at least within a group. We may debate whether the next group should ever take precedence over the previous one, or never, or only some times. This simplification does force layered encodings to use separate tracks for each layer, and to explain the relation between the tracks as part of the application data. Which amounts to pushing complexity out of the transport and back into the application, which I think is a better tradeoff. The tradeoff also simplifies the tasks of the control stream, because it only has to manage precedence between streams, not precedence between objects.
Should be fixed with the PR that
I would add a note that data streams are sent on unidirectional streams, but looks gud.LGTM
It's written: Actually I'd rather say "MUST send a version no higher than their own in forwarded messages" (poor wording, I know, maybe someone can propose better). Indeed, if an intermediary receives an HTTP/1.0 request and passes it as HTTP/1.1, the server will wrongly assume that the client can deal with 1.1 (e.g. chunks) and the message may have to be degraded by the intermediary (such as de-chunking and rely on close only). Furthermore, seeing 1.0 for a server is often an indication of very limited (or possibly bogus) client. For example some intermediaries might avoid compressing or caching when facing HTTP/1.0 messages, and as such it's preferable to let such versions be properly advertised in messages that could be considered as potentially unsafe. Maybe a different wording should be "intermediaries... MUST NOT pass a message showing a version they do not support, and MUST make sure the message always conforms to the advertised version".
I agree with Willy: the underlying requirement is that intermediaries accept and send messages only in versions that they understand (unless acting as a tunnel, I guess). The original text basically means that, even if it is phrased in terms of what value is placed in a specific field rather than as the true requirement.
Discussed at Feb 21 interim; should work on a proposal to loosen this somewhat, at least for intermediaries. SHOULD + context?
OK let's try with that.
but see feedback from NAME
This is a PR that essentially rebases the change proposed in . To me the change proposed by NAME is a clear improvement. I believe that we should merge this.
Regarding The requirement in the last sentence needs to be re-worded, since includes parsing and the client is clearly parsing those header fields and using them.
I would rewrite the paragraph as
Collision with my suggested edits; closing. Please ask to reopen if you still think there's a problem; we can address it in WGLC.
Although it's true that RUSH predates DATAGRAM, streams were an intentional choice as RUSH implementations can leverage standard QUIC library reassembly of messages longer than one MTU.
Absolutely. It's disingenuous to say that RUSH is using QUIC streams because QUIC datagrams didn't exist yet. QUIC streams provide fragmentation, reliability, flow control, and cancellation (error codes). In fact, you could (but shouldn't) use QUIC streams to deliver individual RTP packets without much downside.
The above commits should fix this, please let me know if I got it wrong.