Code change:
This PR adds a status field to the object that allows an relay to indicate an object or group was lost or dropped. It also allows a producer to indicate a object or group will not be produced including the cases for end of group and end of track. The PR probably needs a bit more text on what what producers and relays do but I think this is close enough to get some initial discussion on this solution. Fixes most of
I think I'm happy to use a byte for this. We can always optimize it later in a number of ways if we want.
We can actually save a byte by omitting Object ID except for the first Object in the stream, because this allows the Objects to be sequential. Similarly, we can omit GroupID if the delivery preference is track per stream.
This is a good point. Conceptually we need two enums: One to indicate if it's a Group Fin or Track Fin and one to indicate why there's no payload. There are a lot of ways to encode that, but it's a bit nice, because the first enum is an immutable property of the Object and the latter is sort of a payload alternative.
I've only had a chance to glance, but how does this interact with RESETSTREAM? Putting drop notifications within the OBJECT itself means that an object can only be dropped prior to being written to a stream. This is okay when used in conjunction with stream priorities, but it's not great. Ideally we can renege after committing data to a stream by resetting it. IMO there should be a message indicating which objects were dropped upon a RESETSTREAM.
Notes to myself: Need text on reset. could send object group drop on new stream open an issue if we need this it Way to deal with sending zero length object add an enum for this object is empty Way to indicate end track and I dropped it. perhaps better way than sending same object twice
NAME i did leave the following comment URL
After reviewing fully, yeah I don't think we can decouple drops from resets. When any publisher (including a relay) decides to drop an object, it will also reset the associated QUIC stream. Otherwise, the publisher has to delay writing the object to streams (like TCP low watermark) so it retains the ability to drop it. And the problem with TCP low watermark in MoQ is of course... priorities. Newer content is usually more important, so even if we already have N bytes queued in the QUIC layer, our new content needs to skip the queue. The only way to do that is to actually write to the QUIC stream marked at a higher priority... or reset streams currently queued. The PR doesn't allow either of these as I understand.
I like what Cullen suggested in the editor's meeting. When we want to drop an object, we RESETSTREAM and write a dropped message on a new stream (that won't be reset). Relays will forward this message too. The caveat is that the object may have been successfully delivered despite the so the subscriber has to be prepared to ignore some drop messages.
Sorry I left a PR review a little too fast between meetings and taking care of the newborn. I had a chance to think things through a bit more instead of leaving vague and passive-agressive messages like: I'm warming up to the status in OBJECT, however it needs to be a bitmask. An object can be both dropped, the end of a group, and the end of a track. It's a little annoying to have to process duplicate OBJECTs with different statuses but it's fine I guess. An alternative would be a reliable SUBSCRIBE control stream that contains reliable messages GROUPEND, TRACKEND, OBJECTDROPPED messages. There would be no need to retransmit anything on a RESETSTREAM but the overhead would be marginally higher. ex. GROUPEND would consist of the message type, the groupid VarInt, and a object_id VarInt only. The track would be inferred from the subscribe control stream. The added benefit is that a subscriber would know the size of the group BEFORE receiving the final object; it would know how much is still in flight.
I think Cullen's update addressed this by making "end of group" and "end of track" have different object IDs than the last actual object. Should we further state that only objects with status 0 can be "dropped by a relay"? All the other statuses may be needed by the subscriber, and all should be very small. Would you be ok with moving this PR forward if we split out 0x5 and 0x6 for now?
Note to cullen - remove the two codes. Clean up other comments. Ping Ian when done and ready to merge.
NAME - Can you give this a read and merge if you think it is OK. I did make the change in the PR luke sent to this but it ended up in another PR so github did not see it as done.
I did another read through and it looks good to me !! Thanks Cullen
Individual Comment: This means a relay cannot assume the existence of these marker objects. That's unfortunate, because I'd like the end-of-group marker to help my relay know when to close fanout streams. I don't want to rely on QUIC FIN for this - a publisher may use a QUIC FIN to terminate a stream cleanly (delivering all written objects) as part of a graceful shutdown, without wanting to signal the end of group. In this case, I explicitly don't want to close fanout streams, in case the publisher is reconnecting (make before break). Would making end markers optional for DATAGRAM tracks sufficiently address the audio only use case?
I get your point and need to think about all of that a bit. Lets say the client did send them, would it be a problem if it just crashed without sending them. Would the relay also have a timer or some other way to clean up the state ? Or just wait for subscribe to end ? Just thinking through implications but you raise a good point.
As far as I understand, we currently allow group IDs to have gaps in them, with only requirement for them is that they are monotonically increasing. This presents a problem with caching. Assume I am a relay, and I receive a range request for ; if the origin has never produced group 15, then even if I have the entirety of that range in cache, I would not know that, since 15 would be just missing, and the only thing I can do with that is to request 15 from the origin every time. Possible solutions to this problem: Add some mechanism to indicate that 15 is not there (e.g. by putting the number of previous group in the beginning of every group). Require group numbers to be incrementing by one. I am fan of approach number 2 since it's conceptually simpler, but I think we can make some version of 1 work.
I am not fan of approach 2 since it is limiting and applies to typical media applications only. I think we need to understand that, the application controls the groupIds and they are independent sync points. Catalog should provide that information for the players. Regarding if there is a gap due to object being dropped, that information is identified within the object itself. Let's take an application example where the group numbers increment by 2. So the groups would be 10,12,14,16,18,20 in the above example. Now consider that group 12 was dropped by the publisher , then the relay's cache will be in this form [10, 12 (not sent by the publisher), 14,16,18, 20] Keeping relay unaware of application semantics is something we should allow. Regardless, the end applications know exactly how the groups behave and if there are geps (either due to broadcaster not producing or dropped for other reasons), things will be identified over dataplane as such
NAME can you give a specific use case for how an application could use non-sequential group IDs for some benefit?
Now consider that group 12 was not sent by the publisher , then the relay's cache will be in this form [10, 12 (not sent by the publisher), 14,16,18, 20] The problem I have here is not with 12, the problem here is that the relay cannot distinguish 11,13,15,17,19 being in a state "has not arrived yet" vs state "does not and will never exist", so it will be always forced to fetch those.
The relay should not worry about the semantic structure . Relay needs to fetch only those that have been marked as dropped. If it hasn't arrived, then it will eventually arrive or the upstream will mark it as dropped. In this example, Relay fetches 12 and sends rest from its cache. If relay makes decision on behalf of the application, it will end up fetching things that are not even part of the application ( here the groups 11,13,15,17,19) My proposal is to act on the things you know explicitly and don't guess on the unknown.
I'm confused. How is the relay supposed to know that when being asked for range , it does not need to fetch 15?
Say relay cache is empty, it will ask upstream from range [10-20) .. The upstream publisher will send 10,12,16,18, 20 .. Relay responds with that. OTOH if the upstream provides 10, 12,, 16, 18(dropped due TTL), 20 and if the operation is fetch , the relay will fetch 18 to fill in the gap and once 18 arrives , it will be able to send 18 with others.
Sure, let's assume there are no drops, the first fetch is successful, and it puts 10, 12, 14, 16, 18 in cache. Now the second fetch arrives for the exact same range, . It looks at its cache; it has objects 10, 12, 14, 16, 18, that it can return immediately, but it doesn't know if that's all objects that exist in that range or not (we know that because we know that the previous request was , but it is entirely possible that those objects ended up in cache since they were results of five different fetches), so it has to repeat the fetch for those.
I strongly think we should have sequential groups. Like Victor said, the group ID can be used to detect gaps at the moq-transport layer if it's sequential. I understand the desire to stuff application-specific metadata into fields, but it's a layer violation. Put application-specific stuff in the object payload (ex. timestamps) and put properties useful to relays in the moq-transport layer.
GroupIds are just application sync points and are by definition independent of each other. Adding a correlation between them can be invalid/incorrect/misleading as a Relay is unaware of how application defines its sync points. OTOH objects have dependencies within a group, hence the app decided to send them part of a given group and having relay maintain a state to find out gaps may be useful.
I am also a supporter of incrementally increasing Group IDs. Having such IDs does not break the contract that these points are independent of one-another. It simply means that they occur in a sequence. And being part of a sequence is a fundamental attribute of all MOQT groups, since their parent, a track, represents a temporal flow of data. This example from Victor illustrates the core problem. For anything other than a contiguous sequence of group numbers, a relay can never know if the Groups exist or not and it would always have to make a request upstream to check. It could keep track of prior subscriptions (for example [10..12], [14..16], [18..20]), however if any of those did not overlap then it would be forced to make a fetch for [12..14] and [16..18] in case groups 13 and 17 existed. That is a wasted and convoluted workflow. I tried to think of a use-case where I would want non-incremental group numbers. One example would be to use epoch time as group number. So if I'm making variable length segments, and doing segment-per-group, I might number my groups: A player wanting to seek back in time would need a timeline track to tell it the precise group number it needed. That timeline track could equally map group numbers to epoch time, which would allow me to number my groups incrementally: The meta-point here is Group IDs only define sequence and any other relationship between those sync points can be expressed via an application level timeline track and/or catalog.
temporal flow of data doesn't imply sequential nature of groups either.
Second fetch gets the exact same answer (10,12,14,16,18 since it falls in the range). If there were gaps due to groups being dropped on path or a source deciding to skip a group , those should be marked for relay to see if it needs to satisfy fetch request. I feel it will help us to think on why gaps can occur and those can be explicitly signaled . Relays then in application agnostic way acts on the data in cache.
NAME How does the subscriber know the SUBSCRIBE/FETCH is done? We have a SUBSCRIBE_DONE message with end=20. The subscriber receives 10,12,14,16,18,20 but when does it stop waiting? Is group 19 still in transit? There's no drop notification for it since it doesn't exist.
Subscriber application knows its catalog and deduce the group distribution and find out there will be no grpup 19.
The moq-transport library should tell the application that a subscription has terminated, and not the other way around. Especially since the moq-transport library can't release any state until this optional signal is provided by the application (which might not care about gaps). If the application needs to use a catalog/timeline to detect gaps, then it can use that same catalog/timeline to carry the timestamp that is being shoved into the . Or it can put it in the object payload.
So we're talking about two different fields and I'm going to use disambiguating names: : Increases by 1 each group. Provides ordering and gap detection. : Increases by an application specific amount for each group (no units). Provides ordering. There's a world where both are separate properties (and fields) of a group in the moq-transport layer. Maybe the relay could use the epoch (as a timestamp with units) to perform TTLs, or latency budgets, or fetches. But I think we should punt timestamps and metadata in general to the application for now. I'm also not sure how this would work in practice anyway without an equivalent , but are already sequential.
We define a track (our subscribable entity) as a "sequence of Groups". That is part of our object model. There is no part of a track which does not belong to a group. Therefore, a track consists of a sequence of groups. ! This diagram shows three tracks. The first two tracks are valid, one with temporally contiguous and the other with temporally non-contiguous groups. The third track is not valid, as we don't allow a publisher to produce objects at the same time in the same track which belong to different groups.
this looks like an example constructed incorrectly. None of my points above said overlapping groups Ids. All I am saying is group Ids are independent and don't have to increase in sequence. As you pointed out, both track 1 and track 2 are valid, which i 100% agree. I am proposing we shouldn't force an application to follow what looks like just the track 1 model, since track 2 is also a valid group distribution. MoQT should have necessary things to allow track 1 and track 2, if not, let's define it.
I think you missed the point in my comments . No catalog is not used to define gaps. Gaps are signaled in the data plane
Exactly. I am not arguing either. All I am saying is, when a subscribe done is processed and reported to the application, the application ha enough information to know if there needs to be group 19 or not, from your example. MoQ transport library reports status of what it has seen and only the application knows more about, if group 19 was supposed to be even produced for a given track or not , since that info is in catalog. Please note , this is nothing to do with GAPs. Gaps happens when things are dropped for variety of reasons. Track properties/characteristics (like number of groups, how groupIds are used, group duration) is catalog scoped and not up to for the moq transport to enforce. This is just the principle of separation of concerns across layers.
In the current state, since the moq-transport library cannot detect gaps, it has to immediately report any SUBSCRIBEDONE to the application. OBJECTs may still be received during this state so the library does not drop state associated with the session. The application has to tell the library when the subscription is actually done so it can remove the lookup table entry. With sequential groups and drop notifications, the moq-transport library can buffer the SUBSCRIBEDONE until it has received and flushed all OBJECTs to the application. The application only learns the subscription is done after receiving the last object. This is how transport APIs generally work and how QUIC works itself.
And just to draw inspiration from QUIC itself, it does not allow gaps in the Stream ID or Stream Offset (roughly analogous to groupid and objectid). QUIC could have allowed the application to introduce gaps in either but it just complicates everything.
Not sure how to interepret this in the context of this discussion. QUIC is transport layer protocol and MOQT is application protocol. Also StreamIds and GroupIds are semantically different.
Chair Comment: I think the WG can build the system to support either sequential or non-sequential groups. Several folks have put forward reasoning for why sequential groups provide benefit and/or why non-sequential groups add complexity. NAME as the lead proponent for non-sequential groups, can you state the use case and application benefits. This will help us evaluate the tradeoffs.
The problem is how to resolve this on the cache. Right now, the cache, when it does not have any information about the state of object 15, cannot tell apart the scenario where 15 does not exist, vs the scenario where 15 is missing from the cache because the cache got filled by one client doing a request, and then another doing . It is possible to work this around by putting a "does not exist" kind of entry into cache on FETCH: if I fetch , and get , I can cache " does not exist" from my knowledge of the properties FETCH provides. Outside of being more complex, this also does not solve the entire problem, since our goal is to be able to cachefill from SUBSCRIBEs as well as FETCHes. So non-sequential group IDs inherently makes caching less efficient.
Let's take the example of sequential groups for a similar use-case Fetch 1: 10-14, cache is filled with 10, 11,12,13,14 Fetch 2: 18-20: cache is now filled with 10, 11,12,13,14, 18.19,20 now fetch 3 arrives with 10-20, Cache doesn't know the state of 15,16,17 . So as it needs to do, it will have to ask upstream for . In the case where 15 was never produced for whatever reason, the publisher will need to mark it as such in the response regardless (like ), as this will aid subsequent requests to not repeat the same request. One observation on the cacheFill, When producer sends groups and objects, either as response to subscribe or fetch, the cache fill happens regardless. Let's say when subscription was made , an object was marked as dropped due to congestion. An eventual fetch request can end up replacing that object with proper object if it is still available upstream.
Are you saying that the publisher is allowed to skip group IDs, but it has to explicitly mark groups in the middle as missing? This de-facto makes them sequential.
Not exactly that. This is the gap use-case that had a requirement for players who want to know if a given group that is expected is marked explicitly when source drops, thus allowing player to not wait forever. There are couple of things we are talking here IIUC. Publisher explicitly marking things when it decides to drop. Publisher responding to the request for a non existent things (never produced, permanently gone or dropped) Players knowing not to expect certain groupIds since they know from catalog that the group distribution will not generate such groupIds. Players knowing not to wait for missing things. Cache-fill from either subscribe and/or publish. Relay asking upstream for things that is not its cache Clients and Relays need to be able to process groups out of order. All these can be done or needed regardless of how group numbers are distributed.
Want to clarify that, one needs to request for 15 only one time, if and only if the fetch request asks for it. Also a client who is aware that group 15 will not be generated, per catalog, will not ask for it either. Regardless of the group distribution, relays can be asked for things that it not in its cache. Asking origin for that evaluates to a non existent things ( never produced, not produced since origin decided to drop it or permanently gone from the origin) needs to be updated in the cache. Since a fetch can happen for things that fall in to any of these cases and a relay that doesn't have that information in the cache will need to go back and find out the answer and it needs to do only once.
I want to second this. NAME would allowing groups of size 0 work for you? The problem today is the ambiguity. A subscriber doesn't know if 15 is in flight or will never arrive. If the publisher explicitly tells the subscriber that 15 is empty (or dropped) then that solves things.
yes if there is a reason for a producer to drop entire groups and should be explicitly signaled. "The problem today is the ambiguity. A subscriber doesn't know if 15 is in flight or will never arrive. If the publisher explicitly tells the subscriber that 15 is empty (or dropped) then that solves things."__ We need things to be marked to indicate things when dropped at the publisher. This is needed for reasons pointed out in URL and URL
I don't get what the problem is here. The things the relays cache is the object. I am very concerned that any assumption on groups being sequential are being produced in order are not going to work with the way people want to use complex reference frames and layer in modern codecs like AV1. Of course an given application like youtube might specify for it's usage of moq, everything was sequential. But the moq relays don't need to know this or assume this and keeping them flexible allows for all kinds of interesting media uses cases for media beyond video. I have helped write a relay, and I don't see what problem this is causing for relays.
Ah, my concern was that someone would try to have groups numbered 0, 1000, 2000, or use group ID as a timestamp, in which case you'd have to send hundreds of gap indicators. If that is out of scope, simple "a group is missing" would work.
+1. PR provides aspects of that for the cases when the publisher explicitly drops a group IIRC.
The current approach of allowing any OBJECTs to arrive on any stream makes it impossible for the receiver to know when a group ends. There's a few conditions when you need to know the end of a group: A relay being told to terminate the subscription after group N (subscription hints). A relay knowing when to close a QUIC stream (if it writes a group per stream). A decoder deciding when to start decoding/rendering the next group (ex. GoP). Likewise there's a similar problem of knowing when there's an gap within a group. There's no way to signal that OBJECT X will never arrive, so the receiver should not wait for it.
I strongly advocate for a 1:1 mapping of group to stream. Objects within a group are in order, but there may be gaps. The stream FIN signals the end of a group. The sequence number (or FIN) can be used to determine gaps. An application that doesn't support gaps or skips a group can send STOP_SENDING; something that is not possible with an arbitrary stream mapping.
Alternatively, we could have , and control messages. I think this is a mess, and the receiver would need a corresponding in response to an unrecoverable gap caused by . I don't think we can annotate OBJECTs like NAME once proposed with a GROUPFIN flag because they're lossy. You need some reliable mechanism to signal the end. For example, sending the OBJECT with a GROUPFIN flag over a datagram would be illegal, as would sending a resetstream/stopsending for any stream containing a GROUPFIN object.
Maybe this a duplicate of ?
The issue mentioning Last Object Previous Group is in
I think the best solutions to this is a header on the last object in the group that indicates it is the last object in the group.
Individual Comment: I share Luke's concern above about OBJECT being somewhat unreliable. I think a GROUPFIN message with stronger reliability guarantees would look similar on the wire but work better. eg: Instead of Have This works better than implicit mapping from groups to streams. A relay might need to close a connection with an open stream, but wouldn't want to reset it and wouldn't want the stream FIN to be interpreted as a group FIN. I don't want to put GROUPFIN on the control stream, but that would be the safest. I think the other messages Luke mentioned related to DROP are probably an orthogonal issue.
Agreed, I'm just bringing it up early because stream mapping has massive ramifications on what control messages we'll need in the future. If we can't utilize QUIC's STREAMFIN, STREAMRESET, STOPSENDING, MAXSTREAMS... then we'll need our own, analogous control messages.
you guys have convinced me
As I mentioned during the interim meeting, it makes sense to put that information in the header of the next group. Something like: That information can be used by the receiver to decide when to start playing the new group, and when to wait for more information instead. It is useful: when using "stream per object", to understand how many individual objects are expected for the previous group, when using "stream per group", if group numbers are skipped. The downside is that the relay needs to know the value of previous group and last object before sending the first object of the next group. This may or may not be a problem in practice -- relays may be receiving the information from the origin. If the relay sets an overoptimistic value of previous object, it will have to send some kind of replacement for the skipped objects, such as Which will inform the receiver that the specified object ID and the following ones will not be sent for this group.
Individual ReviewI removed my -1 so if others think it's fine, merge it.I think this fine. Like you, at this point I don't really care much how we spell it. Compared with my proposal in , this is conceptually way simpler (no special rules for each forwarding preference) but is less byte-efficient. I'm happy to go with whatever the group decides.This is a dramatic improvement over the status quo, I think we should merge this without bikeshedding too much about the encoding for now.

Related discussion:
text: and Datagram.  An Object MUST be sent according to its "Object Forwarding Preference", described below. <ins> Object Status: As enumeration used for missing or dropped objects that indicates why data is missing. </ins> Object Payload: An opaque payload intended for the consumer and SHOULD NOT be processed by a relay. <ins> There is also a status field which help subscribers understand what objects it will not receive due to then being dropped, lost, or never be produced including the case where they will not be produced because they are beyond the end of a group or track.  Every object has an associated "Status" that can have following values: 0x0 := Normal object.  The payload is array of bytes and can by empty. 0x1 := Indicates Object does not exist.  Indicates that this object does not exist at any publisher and it will not be published in the future.  This SHOULD be cached. 0x2 := Indicates Group does not exist.  Indicates that objects with this GroupID do not exist at any publisher and they will not be published in the future.  This SHOULD be cached. 0x3 := Indicates end of Group.  ObjectId is one greater that the largest object produced in the group identified by the GroupID. This is sent right after the last object in the group.  This SHOULD be cached. 0x4 := Indicates end of Track and Group.  GroupID is one greater than the largest group produced in this track and the ObjectId is one greater than the largest object produced in that group.  This is sent right after the last object in the track.  This SHOULD be cached. 0x5 := Indicates Object was dropped at a relay.  Sent by a relay indicating the object identified by the ObjectId was dropped. This SHOULD NOT be cached. 0x6 := Indicates Group was dropped at a relay.  Sent by a relay indicating some of the objects in the group identified by the GroupId were dropped.  The Object ID is one greater than last object sent and MAY be zero.  This SHOULD NOT be cached. Open Issue: We could make end of track and end of group just be a status set on the last object sent on the group or track.  This would mean the status could no longer optional and only occur when the payload length was zero as it would be needed in non zero length packets.  It would also have the issue that when a P frame is produced, the software getting data from the decoder often does not know if it is the last P frame until the next frame is encoded and the software gets an I frame.  We do not want a solution where we add a whole frame of latency on the encoder waiting for next frame so it can set the end of group field on the last last object in the group. Any other value SHOULD be treated as a protocol error and terminate the session with a Protocol Violation (session-termination). There are times where some of the status information is redundant with information learned at the lower layer.  For example, when using stream per object, the end of Group is redundant. In most cases messages with a non zero status code are sent on the same stream that an object with that GroupID would have been sent on. The exception to this is when that stream has been reset; in that case they are sent on a new stream.  This is to avoid the status message being lost in cases such as a relay dropping a group and reseting the stream the group is being sent on. </ins> 6.3.2. Every Track has a single 'Object Forwarding Preference' and