Change:
My 2 cents and it’s probably not worth more: I am following the evolution of this draft and while the issues are starting to be well covered this is RG document so while I approve of the changes it would be nice to highlight the research that is still needed. mjm Marie-José Montpetit, Ph.D. EMAIL
Regarding Marie-José's comment: I agree, and I think that we could approach this by trying to find at least one published example of research per category - ideally the "best" per category. This would show, wherever we find something, what the SoA is - research could focus on improving this SoA. Also, it would probably give us some holes - e.g., I don't think there's any much research on (e2e!) FEC "below the transport", especially using multiple CC instances as hinted on in the draft. These holes would let us identify missing research.
You should define what you mean by fairness. I remember endless discussions about fairness in NC with Bob Briscoe amongst others. Marie-José Montpetit, Ph.D. EMAIL
I have corrected most of the comments and issues. I have added two issues to discuss the discussion points opened by Marie-José. How about we merge this and have a new fresh start ? (from which it will be easier for you to create branches, etc.) I guess, looking at the amount of modifications, NAME : may you merge if OK with you?
Ah sorry, I see the comment now, asking me to merge - ok, then I click these 3 "commit"s and then I merge! Doing it now!
Emmanuel made a comment on partial reliability. Should we need a new subsection 6.X to discuss this ?
I don't get this: Emmanuel's comment was about partial order, not partial reliability. Quote: One important thing to consider is reordering and not only reliability alone, we could also consider Partial Order Connection (POC) and Partial Order Transport Service (see Paul Amer works). Spurious retransmission is not only linked to reliability but reordering.
If this issue ends up being about ordering, and not partial reliability: From francoismichel (copied over from a PR): There is something we do not mention here concerning the reliable transport that deliver data in-order : with a reliable transport, the data are delivered in order. I can see two possible cases: Above a single TCP connection: the repair packets won't ever be delivered to the applications if there is a loss occurring before them. They are thus useless as they are delivered once the loss is receovered. Above multiple streams (e.g. QUIC streams or one TCP connection for the source and one for the repair symbols): the repair symbols are less effective: if there is a hole in the data (ex: seq 1 and 3 are received but not 2),you will need to consider 3 a lost too as it won't be delivered to the upper layer.
An answer from me to this: there are transports that can deliver data out-of-order. SCTP can do that, and it's exposed in the interface of TAPS ( draft-ietf-taps-interface ). Multi-streaming is closely related to that: if you take out-of-order message delivery and just add stream identifiers to ensure that, per stream, messages are delivered in order, you have it. This means that FEC could perhaps be applied across streams in case of a multi-streaming transport. I think this just complicates the discussion, let's not go there in the draft.
Thank you for summing up the discussion here. I think we already quite an important number of new sections in this document - we may leave the topic on out-of-order packets on the side and focus on what we have at the moment.
Ok, but do we agree that this is about "partial ordering", not "partial reliability"? In this case, we should probably rename this issue to avoid further confusion.
Text proposed in PR
Should we consider the impact of cross trafic that uses congestion-controled transport protocol?
done in PR
There should be a discussion on the amount of redundancy of the FEC e.g. FEC above transport : including redundancy reduces goodput for large files transfers e.g. FEC in transport : including redundancy reduces goodput for large files transfers but can be adapted depending on the CWND size e.g. FEC below transport : including redundancy adds congestions without reducing goodput
From Dave Oran (IETF108): One thing it might be good to cover is how to couple increasing transmission rate with increasing/decreasgin coding rate. It may be interesting to use increasing coding rate as a "probe" to more safely increase transmission rate. Has this been looked at?
Done in PR
More related work, examples of FEC research work? Papers? This kind of thing? (I agree, just clarifying if this is what this issue is asking for)
Started in PR
The title of the document is "Coding and congestion control in transport" but we only present a "FEC Channel" that only carries repair packets and some signaling about the repaired packets. I like the idea of two separated channels a lot, but I think we also should integrate source packets as part of the FEC channel, and maybe rename the FEC channel to "Coding Channel" and "packets" to "symbols" when we talk about the coding channel ? Here is what I propose concretely: The CC channel stays unchanged, it carries packets The FEC channel becomes a Coding channel and carries symbols: Source Symbols and Repair Symbols. I think the approach there is more general: one network packet can contain several Source/Repair Symbols (similarly to the coding-for-quic draft). I think this helps to isolate both CC and FEC channels from each others as the FEC/Coding channel only plays with Symbols while the CC channel plays with packets. When I read the current design, it makes me think that the FEC channel needs the source packets from the CC channel to decode the received repair packets but this is shown nowhere in the graph. By considering different entities (packets VS symbols that are in fact a view on the sent/received data), we decorrelate more intuitively the FEC and the CC channels. As a consequence, we will only talk about symbols when it concerns the Coding channel and packets when it concerns the CC channel. What do you think ? I do not think this is an urging change by the way.
Hi, Coding channel refers to physical layer and when combined with the term symbol, it also refers to physical operations which are not those at packet level. We stated in RFC8406 that Generally speaking, a Packet is the unit of data that is sent in the Packet Erasure Channel, while a Symbol is the unit of data that is manipulated during the encoding and decoding operations. So we allow the use of symbol only when it corresponds to non yet transmitted data. As soon as transmitted, it becomes a packet. EL
Hello, symbol, it also refers to physical operations which are not those at packet level. Hmm, in that case, renaming the channel this way would make it more confusing I agree, sorry for that then :-) transmitted data. As soon as transmitted, it becomes a packet. I think the definition I proposed here still fits with the symbol definition you described though. A packet can contain more than 1 source/repair symbol(s) and the source symbols can be different than what is actually sent on the wire (e.g. source symbols can contain padding that is not actually sent on the wire or can contain metadata that are not sent on the wire either). So I still think that it matches more the definition of symbols than packets. All the real packets received by a protocol can be processed and transformed into source symbols before being passed to the FEC Channel. (sorry in advance if I have misconceptions on this matter)
Yes in that context it's fine and we stick to RFC8406.
Done in PR
DO: Fine, but you miss the opposite case: how many repair packets are you sending that turn out to be useless? It's a matter of measuring the usefulness of repair traffic. It gives an extra degree of freedom as you may want to play at the congestion control level or at the coding rate level. NK: Good point, that's the kind of topic we need to gather in the document. There are several corner cases that are very important. This point is about how you add coding inside congestion control. With this claim, we want to be generic.
I think this document ought to remove the confusion between coding and congestion control that folks seem to have. So: "It gives an extra degree of freedom as you may want to play at the congestion control level or at the coding rate level." sounds like it's based on this confusion. And: "This point is about how you add coding inside congestion control." doesn't seem right either.
Indeed - the introduction of the coding and congestion channels could help. I may need to be reflected in the rest of the document.
Maybe we could emphasize on the fact that coding is a reliability mechanism and is distinct and separated from the loss detection and congestion control mechanisms. If all we care about is bandwidth in a general unicast scenario, sending a-priori repair packets should not help more than Selective-Repeat ARQ. In the best case, it should perform as well as a SR-ARQ if the packet is lost (without additional gain), and will simply waste bandwidth compared to SR-ARQ if the packet is not lost. I can identify one case where doing FEC can help for bandwidth: when the buffer are full or very limited and cannot be in-order delivered to the application because of head-of-line blocking. Sending FEC could help to unblock the peer's receive buffer sooner. But this has nothing to do with congestion control.
I think this kind of discussion should be in the document.
I agree with NAME 's idea of unblocking the receiver. Also, in the other issue, he mentioned the big disadvantage (delay) coming from tail losses. These things should now be added to the document - now it's structured such that it could be added better, in the right place. Other thoughts that were brought up in discussions, and should go into the doc - both are about the FEC-below-transport case: 1) ECN: From NAME : There is more detail about possibilities to use ECN for FEC-below-transport in Emmanuel's presentation: URL (slide 4) 2) special environments: and we had this discussion in a PR related to FEC-below-transport: NAME wrote: and I answered: .... and about the FEC-above-transport case slide 2 reminds me that this is obviously useless if the transport repairs all losses, as with TCP! This is worth mentioning in the FEC-above-transport case - we could mention that unreliable congestion-controlled transports (e.g. RMCAT cc's, or DCCP) are better to consider here than fully reliable ones such as TCP.
I am unsure where this issue is going and what we can integrate in the draft then. I would say that identifying useless repair packets is the work of the reliability mechanism (there is a clear similarity with spurious retransmissions detection) and in my opinion has nothing to do with congestion control, so what can we integrate in the draft ?
I think this discussion is covered in PR
Thanks for the PR ! :-)There are two small suggestions that I made... not sure they're committed yet? Else just press "commit" 3 times please :)

Discussion:
carries source packets from a sender to a receiver, and packets signaling information about the network (number of packets received vs. lost, ECN marks, etc.) from the receiver to the sender.  The <del> Forward Erasure Correction channel carries repair packets (from the </del> <ins> Forward Erasure Correction channel carries repair symbols (from the </ins> sender to the receiver) and potential information signaling which packets have been repaired (from the receiver to the sender).  It is worth pointing out that there are cases where these channels are not

including CC): refers to the rate at which a congestion control decides to transmit packets, based on 'network information'. <del> 'signaling repaired packets' (input control plane for the FEC): </del> <ins> 'signaling repaired symbols' (input control plane for the FEC): </ins> refers to the information a FEC sender can obtain from a FEC receiver about the performance of the FEC solution as seen from the receiver.

'coding rate' (output control plane for the FEC): refers to the coding rate that is used by the FEC solution. <del> 'source and/or repair packets' (data plane for both the FEC and the CC): refers to the packets that are transmitted.  There can only be source packets (if the coding rate is 0), repair packets (if the solution decides not to send the original source packets) or a mix of both. </del> <ins> 'source and/or repair symbols' (data plane for both the FEC and the CC): refers to the data that is transmitted.  The sender can decide to send source symbols only (meaning that the coding rate is 0), repair symbols only (if the solution decides not to send the original source packets) or a mix of both. </ins> The inputs to FEC (packets to work upon, and signaling from the <del> receiver about losses and/or repaired packets) are distinct from the </del> <ins> receiver about losses and/or repaired symbols) are distinct from the </ins> inputs to CC.  The latter calculates a sending rate or window from network information, and it takes the packet to send as input, sometimes along with application requirements such as upper/lower

3.2. <del> While CC is in principle independent of other transport functions such as reliability, we note that CC is often embedded in reliable transfer protocols (e.g.  TCP).  This approach may require that the transport protocol does not implement a fully reliable data transfer service (e.g., based on lost packet retransmission), otherwise this could lead to spurious retransmissions.  UDP is an example of a protocol for which this approach is relevant. </del> <ins> The advantage of this approach is that the FEC overhead does not contribute to congestion in the network.  When congestion control is implemented at the transport layer, the repair symbols are sent within what the congestion window allows.  This approach can result in improved quality of experience for latency sensitive and non reliable applications. </ins> <del> The advantage of this approach is that the FEC does not contribute to adding congestion in the network.  When a congestion control is implementedd at the transport layer, the repair packets are sent within what the congestion window allows. </del> <ins> This approach requires that the transport protocol does not implement a reliable data transfer service (e.g., based on lost packet retransmission).  UDP is an example of a protocol for which this approach is relevant.  For reliable transfers, the usage of coding does not guarantee better performance and would mainly reduce goodput for large files transfers. This discussion section is extended in sec:fairness. </ins> 4. 4.1. fig:fec-in presents an architecture where FEC is within the <del> transport.  The repair packets are sent within what the congestion </del> <ins> transport.  The repair symbols are sent within what the congestion </ins> window allows, such as in CTCP. 4.2. The advantage of this approach is that it can enable conjoint optimization between the CC and the FEC.  Moreover, the transmission <del> of repair packets does not add congestion in potentially congested </del> <ins> of repair symbols does not add congestion in potentially congested </ins> networks but helps repair lost packets (such as tail losses). <del> The drawback of this approach is that it may not result in much gains as opposed to classical retransmission mechanisms and it costs bandwidth that could have been exploited to transmit source packets. The coding ratio needs to be carefully designed. Examples of the solution would be sending repair packets when there is no more data to transmit or preferably send repair packets instead of the following packets in the send buffer. </del> <ins> For reliable transfers, including redundancy reduces goodput for large files transfers but the amount of repaired symbols can be adapted, e.g. depending on the congestion window size.  There is a trade-off between the cost in bandwidth that could have been exploited to transmit source packets and the advantages that are brought by transmitting repair symbols (e.g. unlocking the receive buffer if this is limiting).  The coding ratio needs to be carefully designed.  For small files, sending repair symbols when there is no more data to transmit or preferably sending repair symbols instead of the following packets in the send buffer could help to reduce the transfer time.  This is because it could avoid a silent period between the transmission of the last packet and firing the retransmission of lost packets. This discussion section is extended in sec:fairness. </ins> 5. 5.1. <del> fig:fec-below presents an architecture where FEC is below the transport.  The repair packets are sent on top of what is allowed by the congestion control. </del> <ins> fig:fec-below presents an architecture where FEC is applied end-to- end below the transport layer, but above the link layer.  Note that it is common to apply FEC at the link layer, in which it contributes to the total capacity that a link exposes to upper layers.  This application of FEC is out of scope of this document.  The repair symbols are sent on top of what is allowed by the congestion control. </ins> 5.2. <ins> In this case, including redundancy adds congestion without reducing goodput but leads to potential fairness issues.  Indeed, the effective bitrate is higher than the CC's computed fair share due to the sending of repair symbols and the losses are hidden to the transport which may cause a problem for loss-based congestion controls, but is not a problem for delay-based ones. </ins> The advantage of this approach is that it can result in performance gains when there are persistent transmission losses along the path.

designed. Examples of the solution could be adding a given percentage of the <del> congestion window as supplementary packets or sending a given amount of repair packets at a given rate.  The redundancy flow can be </del> <ins> congestion window as supplementary symbols or sending a given amount of repair symbols at a given rate.  The redundancy flow can be </ins> decorrelated from the congestion control that manages source packets: <del> a secondary congestion control could be introduced underneath the FEC layer.  The separate congestion control mechanisms could be made to work together while adhering to priorities, as in coupled congestion control for RTP media I-D.ietf-rmcat-coupled-cc.  Another possibility would be to exploit a lower than best-effort congestion control RFC6297 for repair packets. </del> <ins> a separate congestion control could be introduced to manage the amount of repaired packets to transmit on the FEC channel.  The separate congestion control instances could be made to work together while adhering to priorities, as in coupled congestion control for RTP media RFC8699 in case all traffic can be assumed to take the same path, or otherwise with a multipath congestion window coupling mechanism as in Multipath TCP RFC6356.  Another possibility would be to exploit a lower than best-effort congestion control RFC6297 for repair symbols. This discussion section is extended in sec:fairness. </ins> 6. <del> This memo includes no request to IANA. </del> <ins> The objective of this section is to further detail some aspects that have been expressed in previous discussion subsections. 6.1. The objective of this section is to further discuss how adding coded flows impacts on non-coded flows when they share the same bottleneck. This section assumes that the non-coded flows respond to congestion signals from the network. 6.1.1. The addition of coding within the flow do not impact on how the coded and non-coded flows interact.  This interaction would mainly depend on the congestion controls embedded in each host. 6.1.2. The addition of coding within the flow may impact on the congestion control mechanism and hide congestion losses.  Specific interaction between congestion controls and coding schemes can be proposed (see subsec:cc-recov-interaction, subsec:cc-nc-interaction and subsec:cc- useless-interaction).  If no specific interaction is introduced, the coding scheme may hide congestion losses to the congestion controller and the description of subsub:fec_below may apply. 6.1.3. In this case, the coding scheme may hide congestion losses to the congestion controller.  There are cases where this can drastically reduce the goodput of the non-coded flows.  Depending on the congestion control, it may be possible to indicate that a packet has been recovered, e.g. using ECN flags, to reduce the impact on the non-coded flows (see subsec:cc-recov-inter-below). 6.2. The objective of this subsection is to destribe potential interactions between the congestion control and the recovered symbols. 6.2.1. The congestion control may not be able to differentiate repair symbols from actual source packets.  The relevance of adding coding at the application layer is related to the needs of the application. For real-time applications, this approach may reduce the amount of retransmission.  The usage of a non-reliable transport is more adequate in this case. 6.2.2. If the two FEC and CC channels are decoupled, the endpoint may exploit specific protocol for each of the channel.  The channels may be coupled and one single protocol may be exploited.  In both cases, the receiver can differenciate source packets and repair symbols. The receiver may indicate the amount of source packets that have been received and the amount of repaired symbols that were actually useful in the recovery process of packets. QUIC may require a specific type of frame to transport such information between receiver and sender. 6.2.3. The congestion control may not know what is going on in the network underneath and whether a coding scheme is introduced or not.  The congestion control may behave as if no coding sheme is introduced. The only way for a coding channel to indicate that symbols have been recovered is to exploit existing signaling.  An example would be a receiver to indicate that a packet has been recovered by using ECN signals of TCP. 6.3. This section discuss to what extent interaction between the congestion control and the coding rates are possible. 6.3.1. The coding rate applied at the application layer mainly depends on the available capacity given by the congestion control underneath. Adapting the coding rate to the minimum required data rate of the application may help to reduce packet losses and improve quality of experience. 6.3.2. In this case, there is an important flexibility in the trade-off between reducing goodput, helping recovering from transmission and congestion losses, being fair in respect with non-coded flows.  Such as explained in subsec:cc-recov-inter-in, the receiver may be able to indicate the sender the amount of packets that have been received or recovered.  The sender may exploit this information to tune the coding ratio.  As one example of flexibility of this case, coupling an increasing transmission rate with an increasing or decreasing coding rate could be envisioned.  A server may use increasing coding rate as a probe of the channel capacity and adapt the transmission rate of the congestion control. 6.3.3. The coding rate in this case can be tuned depending on the amount of recovered symbols and the rate of which data are transmitted by the sender.  The coding scheme is not aware of the congestion control implementation, making it hard for the coding scheme to apply the relevant coding rate. 6.4. There are cases where useless repaired symbols may be transmitted. These have an impact on the network load and may reduce the goodput of the flow without concrete gains. 6.4.1. In this case, the discussion depends on application needs.  The only case where adding useless repaired symbols does not result in reduced goodput is when the application needs a limited amount of goodput (e.g.  VoIP traffic).  In this case, the useless repaired symbols would only have an impact on the amount of data generated in the network. 6.4.2. The sender may be able to exploit the information given by the receiver to reduce the amount of useless repair packets and the resulting goodput reduction. 6.4.3. In this case, the useless repaired symbols only have an impact on the load of the network without actual gain for the coded flow. </ins> 7. <ins> This memo includes no request to IANA. 8. </ins> FEC and CC schemes can contribute to DoS attacks.  This is not specific to this document.