Change:
Thanks so much for doing this! If you incorporate my comments, you've written exactly what I wanted to do, you've just taken the work from me :)
SD: We discussed in many RFC congestion control. I encourage you to include scalable congestion control and the IETF history on the reaction to ECN signals. EL: Good point.
I was talking to Gorry Fairhurst afterwards, and Gorry told me what I SHOULD have said, was that the sender should perform whatever its response to congestion would be - if it's an old-style sender, treat recovery as loss, but if it supports scalable congestion control (L4S is one example), it doesn't have to treat recovery as loss - it just needs to do something appropriate.
Gorry's point sounds like "the congestion control reaction is what it is, it has nothing to do with coding", and I'd agree with that. I would therefore not talk much about different types of cc. in the document, as this may just add more confusion.
I think that this is covered by the current version of the document. "The server MUST be able to detect the "repaired packet" signal. The base solution does not describe how the server reacts to such signal."
I may have forgotten things over the holidays, so I'm not sure how clear the document is on that ... but I don't like the sentence that you quote because it sounds like "repaired packet" is a different kind of signal than "lost (and not repaired) packet". For congestion control, that distinction shouldn't exist. I see coding and CC as two separate layers, benefiting from different signals.
If I remember well, we formulated this sentence this way : 1) to start reactions and discussions as it is quite a strong claim as Nico said it during the meeting 2) adding the "repaired" signal does not seem to me to interfere with the other signals: it is just an additional signal that can be ignored by the congestion control. A "repaired" packet does not imply a lost packet (the repair could be due to re-ordering), a lost packet will be determined by the loss detection mechanism. So in my opinion, we can have different cases for repaired packets : a lost and repaired packet: the packet has been both repaired and detected as lost (the loss detection may happen before of after the loss detection, it does not matter). Even if the packet is repaired, it can still be marked as lost by the loss detection mechanism because "repair" does not mean the same as an acknowledgement. a non-lost repaired packet (re-ordering implied the repair of the packet but the original packet has been received afterwards) A loss-based congestion control will likely attach itself to the "lost packet" event that will be triggered regardless of the fact that the packet has been repaired, so I don't think it will be perturbated in any way by the "repaired packet" event. In my opinion, a congestion control mechanism could also attach itself to the "repaired" event to do whatever it wants with this additional signal (I have no idea on what it could do with this signal, but a new congestion control could define some heuristics based on the different signals occuring during the connection). I'm saying this with QUIC in mind, it may be more difficult to not mix the "repaired" and "lost" events with TCP (but is seems doable to me a-priori).
I think I'll insist on this point. This is exactly the clarity that's missing in this document: because coding and congestion control are discussed as a conflated whole, it sounds as if, e.g., the difference between "lost but repaired" and "lost but not repaired" might matter to a congestion control mechanism. You even suggest that a new control could do something useful with it; why?? What value does it add, when a congestion control mechanism tries to find the right sending rate based on how full the network is? The two are really entirely separate and should be discussed and presented as such. Implementations may produce the signals you describe, such that their senders can then pick them apart to do what is most reasonable; that doesn't change the fact that conceptually, these are completely separate and discussing them as a combo makes it all confusing and even misleading. This is exactly the confusion the NWCRG has been dealing with for so long. Let's please clear this up once and for all and split these beasts.
I think we need to be sure about the current scope of the draft then. The main idea we wanted to bring with the other guys (correct me if I'm wrong) was to avoid interferences between the coding and congestion control mechanism, especially avoiding a receiver that simply ACKs recovered packets, making the sender think that they have been successfully received. I agree with you, defining a "repaired" signal may lead to think that we consider those two mechanisms as whole. So why defining a "repaired" signal instead of saying that "acknowledging a recovered packet should not be done" ? We discussed this with NAME and NAME and we concluded that defining the "repaired" signal seemed easier to incorporate within a protocol such as TCP compared to the other solution that requires major design changes. With QUIC, we can incorporate the two solutions without any problem. I agree that the current state of the draft is misleading I am OK to change that. We need however to keep in mind the feasibility of the solution on existing protocols. When we started to elaborate the draft with NAME and NAME (again, correct me if I'm wrong) and until now, we decided that determining what the congestion control behaviour should be was out-of-scope for the document. The scope was just the following: we should not hide the recovery/loss signal so that the already existing congestion control mechanisms react the exact same way to losses whether there is coding or not. For the rest, we did not take any decision. This is why you will see many different behaviours described in the current section 6. of the draft (e.g. 6.2.). Until now, we never discussed whether these solutions are good or bad, as they are congestion control behaviours and we are working on the coding component (more specifically what signalling should be done between coded hosts). We just presented them. Because the approach of the document was to avoid interferences between the coding and congestion control mechanisms, not to decide what a congestion control should do or what set of signals it should or should not use. But don't get me wrong, I am not saying that using the "repaired" signal for a congestion control is a good idea. I am fine with the idea of recommending to not use such a signal in congestion controls, but to me, it redefines the scope of the document and might imply other groups than the nwcrg.
I think what I'm suggesting is less problematic than you may believe. To me, some parts of the draft are currently messy because of how they mix the two things - e.g. fig. 2 really hurts my feelings, as a cc. person :) Don't get me wrong, I agree with many things in the draft, and with most of what you say in your comments here too - e.g. this: "especially avoiding a receiver that simply ACKs recovered packets, making the sender think that they have been successfully received." is definitely an important goal! This statement from your comment best describes why we seem to have a disconnect: " I agree that the current state of the draft is misleading I am OK to change that. We need however to keep in mind the feasibility of the solution on existing protocols." because what I'm proposing changes implementation feasibility in no way. This is just conceptual - how we talk about things. Coding can (sigh... if it MUST... ) fit into TCP because TCP does error control, not because TCP does congestion control. I can see one useful input from coding to cc: an "importance" (or urgency?) associated with a data block. I can see useful inputs from cc to coding: network measurements (which is what cc carries out). That's how these things should be described IMO - conceptually separate blocks with interfaces. Implementations can definitely build a compound "repaired" signal which has a meaning to both blocks at the same time, and I don't object to giving implementation examples or advice after introducing things cleanly and separately. But these, concept and implementation, are also two separate things. As for the scope, that's nothing to worry about. This is not about telling a cc. mechanism what to do - it's about explaining the relationship between coding and cc, and that won't change if we re-structure things as I suggest. I'm beginning to think that I should just make a PR where I update everything the way I think it should be, and then we can have a much more focused discussion? Would that be ok? (next week or so... probably not before)
100% agree. I'm beginning to think that I should just make a PR where I update everything the way I think it should be, and then we can have a much more focused discussion? Would that be ok? (next week or so... probably not before) Perfect for me, I do not plan to modify the draft during that time anyway.
+1 I think that the separation of the "congestion" and "coding" channels can help in clarifying there is a difference here. +1 We need to make that clear in the document. (next week or so... probably not before) Ok - I may do some update on master branch before next week
Just to be clear "lost but repaired" is an erased packet rebuilt at destination, erased means it thus can be either a reordered or a lost packet.
Okay, but then there's also the case of "lost but NOT repaired", which is also an erased packet, i.e. either reordered or lost. These two cases are, and should absolutely be, the same for a congestion controller. This is why I don't like feeding "repaired" into a CC.
OK I think I see the problem : you are on the CC-side and we are on the NC-side :) First, I recall we agreed that there are two different signaling paths: one for CC one for NC. A lost but not repaired packet is an erased packet and we don't have any supplementary info at the sender side to provide because the CC will provide it. A lost but rebuilt is a lost packet lost or reordered and we must provide the info at the sender side because the CC won't provide it.
Oh yeah, I see that you (NCers) need the signal! But it needs to be separate from the CC signal. I'm glad to see that you say this was agreed - and the early part of the draft reflects that, but later in the draft (from section 6) things get really mixed up, which is why propose to remove sections 6 and 7.
Not only, when I say "/A lost but rebuilt is a lost packet lost or reordered and we must provide the info at the sender side because the CC won't provide it./" means that the NC receiver needs to alert the sender CC that a lost or a reorder (following the CC flavor used) occurred. This allows the sender to react in a compliant manner as TCP will do. One possible solution would be the NC receiver marks ECN when a packet is rebuilt to inform the sender as explained in my previous NWCRG talk : URL
That's very implementation oriented. Of course, if you want to hack your stuff into TCP and leverage TCP ACKs yet convey that CC. should reduce its rate, you can use tricks like ECN. I would prefer to first discuss these things as entirely conceptually separate - and then, once it's all neat and clean, discuss possible implementation strategies such as this one.
Agree, I showed this just to be sure we lay on the same thread and that all of us (whatever the point of view is from CC or NC) are speaking the same language.
ACK!
One question still remains, what do we do with section 6 which is also implementation oriented and cross-layer. As far as I understand your point, we should not speak or propose such cross-layer schemes as both layers : CC and NC should remain independent right ? Furthermore I'm not happy with the beginning "Delay-based congestion controls ignore packets that have been repaired with coding.". If we lay on the same logic, delay-based should not ignore these packets as a rebuilt one, following the code used and (n,k) length might artificially increase the e2e delay. In that case, one could argue that a rebuilt packet implicitly signal a rebuilt packet to the sender as the delay increase, but this really depends on the kind of code used and its parameters. The more I read this section the more I think we should reconsider it.
cross-layer schemes as both layers : CC and NC should remain independent right ? Yes - I think we CAN then have a section (a new one! I would really rather just dump the current text of sections 6 and 7) that VERY carefully discusses how to implement things together. But then there would have to be text saying that this is implementation only, still the signals are conceptually separate, something like that. For the next version of the draft, I wouldn't even do that but just clean things up (i.e., remove these sections). controls ignore packets that have been repaired with coding.". If we lay on the same logic, delay-based should not ignore these packets as a rebuilt one, following the code used and (n,k) length might artificially increase the e2e delay. In that case, one could argue that a rebuilt packet implicitly signal a rebuilt packet to the sender as the delay increase, but this really depends on the kind of code used and its parameters. You should look at my pull request. Let's see if this URL survives copy+pasting: URL Here, this paragraph now begins with: "[MICHAEL: I would remove this section entirely: it only adds to confusion." :-)
Got all your comments from sections 6-7 and others, I think I see what you mean. Let's restart first considering this comment -> [MICHAEL: So far, it hasn't been clear that we're going to describe a "solution". A solution to which problem? I thought we're only discussing how FEC and CC relate.] Let's re-discuss only how FEC and CC relate, this is what we attempted to do but may be badly.
So I guess the way ahead is to discuss - in the PR - what changes to the PR are desired. If I don't hear any more details by next week, I'll submit an update to the PR where, instead of commenting that sections 6 and 7 should be removed (and why), these sections are gone. What do y'all think about the new diagram, on page 4, and the two paragraphs just below it? This captures how I think about things. Mainly, my take-away is that yes, there may be meaningful interaction between FEC and CC, but no, it's not obviously a matter of sharing the "raw inputs". Instead, e.g. (as one example I'm mentioning), unequal error protection can be an input to CC, as a result of what a fountain code produces. ( an example: chapter 8 of this Ph.D. thesis: URL ... yes, this is me showing off a little, trying to make the point that I'm not COMPLETELY coding illiterate :-) only 90% ! )
There is a fundamental difference between UEP and what we are targeting here, the application of UEP is similar to AL-FEC so there is no interaction at all with a CC as both work at the application layer and I think nothing really new to discuss. Furthermore, UEP needs information about the flow data packet itself which is not our case as we are flow agnostic. In brief, this draft does not tackle AL-FEC. The problem is the use of FEC, RLNC, Tetrys or Sliding Window Code (RFC8680 and 81) below or inside the transport layer. The diagram page 4 shows such AL-FEC application (BTW I would change "information about repaired block" by "information about repaired ratio" to be generic and compliant with SW codes). Consider that you reverse both blocks FEC and CC in your figure, do you see what happens in that case ?
This is getting more interesting. About UEP: interaction at all with a CC as both work at the application layer and I think nothing really new to discuss I think that UEP can assign different priorities to blocks of data, such that they can be more or less important, which could lead to a CC decision to treat some of them with LBE, for example? (RFC8680 and 81) below or inside the transport layer. while I'm admittedly thinking "above". But I placed them next to each other to avoid having to make a layering decision, as it's about separating functionality more than it is about which one is on top of which (which strikes me as more of an implementation decision and less conceptual). "information about repaired block" by "information about repaired ratio" to be generic and compliant with SW codes ACK, will do! About the diagram on page 4: both blocks FEC and CC in your figure, do you see what happens in that case ? No, I don't get it - the idea was to show them side by side so they don't convey a sequence of operation. So in my interpretation, changing their position won't change anything. Or what do you mean?
I think I get a good take-away from this conversation: I keep talking about separating functionality, you keep talking about layering. And, I think we all agree that we want a section discussing implementation possibilities. So, my proposal is to discuss layering there, regarding the transport layer where CC is usually done. 1) FEC above transport, 2) FEC inside transport, 3) FEC below transport. Then we can have a discussion about various issues in this section, but this way it's a clean framework for such text.
I like your proposition of layering. I think that the problem of the first version of the draft was that we tried to consider everything at once. This should be a lot more clear with such a séparation.
Great! I wanted to do it now but I'm running out of time... early next week, promise!
Le 17/01/2020 à 09:22, mwelzl a écrit : Perfect ! I think we start to converge.
Sorry for my late answer - I was sick at the end of the week and could hardly manage all I wanted to deal with .. I caught up with the discussions and think there are some things that we all agree on. I will then merge the pull request so that we can separately work on the document.
I have tried to clean and reorganize the document based on what I think we agree on. See PR Let me know what you think. In case you agree, I guess I can merge it with master so that we have a new basis to go on with.
Once PR is merged to master, I propose to close this issue.
GREAT, thanks!!!

Discussion:
Abstract <del> This document is the product of the Coding for Efficient Network Communications Research Group (NWCRG).  The scope of the document is end-to-end communications: coding for tunnels is out-of-the scope of the document. </del> Coding is a reliability mechanism that is distinct and separated from the loss detection of congestion controls.  Using coding can be a useful way to better deal with tail losses or with networks with non- congestion losses.  However, coding mechanisms should not hide congestion signals.  This memo offers a discussion of how coding and <del> congestion control interact. </del> <ins> congestion control can coexist. </ins> <del> 1. </del> <ins> This document is the product of the Coding for Efficient Network Communications Research Group (NWCRG).  The scope of the document is end-to-end communications: coding for tunnels is out-of-the scope of the document. </ins> <del> This document represents the collaborative work and consensus of the Coding for Efficient Network Communications Research Group (NWCRG); it is not an IETF product and is not a standard. </del> <ins> 1. </ins> There are cases where deploying coding improves the quality of the <del> transmission.  As an example, the server may hardly detect tail losses that impact may impact the application layer [MICHAEL: I don't understand this.  Why would it not detect them?  And why is detecting them the problem?].  Another example are networks where non- congestion losses are persistent and prevent a sender from exploiting the link capacity.  RFC5681 defines TCP as a loss-based congestion control; because coding repairs such losses, blindly applying it may easily lead to an implementation that also hides a congestion signal to the sender.  It is important to ensure that such information hiding does not occur. </del> <ins> transmission.  As an example, it may take time for the server to detect tail losses and this would impact the experience of applications using short flows.  Another example are networks where non-congestion losses are persistent and prevent a sender from exploiting the link capacity. </ins> Coding is a reliability mechanism that is distinct and separated from <del> the loss detection of congestion controls.  This memo discusses simple best practices on how coding and congestion control mechanisms could coexist.  One objective is to help the research community </del> <ins> the loss detection of congestion controls.  RFC5681 defines TCP as a loss-based congestion control; because coding repairs such losses, blindly applying it may easily lead to an implementation that also hides a congestion signal to the sender.  It is important to ensure that such information hiding does not occur. This memo offers a discussion of how coding and congestion control can coexist.  One objective is to help the research community </ins> consider congestion control aspects when proposing coding solutions. <del> The proposed recommendations apply for coding at the transport or application layer.  Coding for tunnels is out of scope for the document. </del> <ins> This document represents the collaborative work and consensus of the Coding for Efficient Network Communications Research Group (NWCRG); it is not an IETF product and is not a standard.  The proposed recommendations apply for coding at the transport or application layer.  Coding for tunnels is out of scope for the document. </ins> 2. fig:sep-channel presents the notations that will be used in this document and introduce the Congestion Control (CC) and Forward Erasure Correction (FEC) channels.  The Congestion Control channel <del> carries data packets (from a server to a client) and a potential </del> <ins> carries data packets (from a server to a client) and potential </ins> information signaling the packets that have been received (from the client to the server).  The Forward Erasure Correction channel carries coded packets (from the server to the client) and a

Inside a host, the CC and FEC entities can be regarded as conceptually separate: <del> As this diagram shows, the inputs to FEC (data to work upon, and signaling from the receiver about losses and/or repaired blocks) are distinct from the inputs to CC.  The latter calculates a sending rate or window from network measurements, and it takes the data to send as input, sometimes along with application requirements such as upper/ lower rate bounds, periods of quiescence, or a priority. </del> <ins> fig:sep-entities shows a example where FEC is on top of the CC, but interactions between the FEC and CC are possible.  In this example, the inputs to FEC (data to work upon, and signaling from the receiver about losses and/or repaired blocks) are distinct from the inputs to CC.  The latter calculates a sending rate or window from network measurements, and it takes the data to send as input, sometimes along with application requirements such as upper/lower rate bounds, periods of quiescence, or a priority. </ins> It is not clear that the ACK signals feeding into a congestion control algorithm are useful to FEC in their raw form, and vice versa

algorithm.  However, there can be meaningful other interactions (indicated by the horizontal double arrow) between the two entities, usually as a result of their operation rather than by relaying their <del> own raw inputs.  For example, the network measurements carried out by CC can yield a longer-term statistical measure such as a loss ratio which is useful input for a FEC coding scheme.  Similarly, unequal error protection using fountain codes can be used to assign different priorities to blocks of data, and these priorities can be honored by a CC mechanism. </del> <ins> own raw inputs. </ins> 3.

added at the server and client end points.  The discussions should then consider fairness with non-coding solutions. <del> 4. </del> <ins> 3.3. </ins> <del> The solution can be described as follows: [MICHAEL: So far, it hasn't been clear that we're going to describe a "solution".  A solution to which problem?  I thought we're only discussing how FEC and CC relate.] </del> <ins> sec:notations presents that FEC and CC can be seen as two separate channels.  In practice, implementations may mix the signals that are exchanged on these channels.  This document discusses how FEC and CC relate.  [NK: We may want to later add that we provide recommendations - but for the moment, we may just list the server side coding solutions in the layering approach without providing recommendations for each case] </ins> <del> The client indicates to the server that one or multiple packets have been repaired using a coding scheme.  The "repaired packet" signal does not guarantee that the packet actually needed to be repaired, since it could have been delayed but not lost. </del> <ins> 4. </ins> <del> The server must be able to detect the "repaired packet" signal. The base solution does not describe how the congestion control reacts to such signal. </del> <ins> fig:fec-above illustrates the FEC above the transport case. </ins> <del> The rationale behind this solution is the following: the server has more information on the congestion status and the application characteristics.  Moreover, congestion control should not be split between multiple entities, as otherwise non-optimized decisions may be taken. </del> <ins> fig:fec-above presents an architecture where FEC is on top of the transport.  The coded packets are sent within what the congestion window allows. </ins> <del> The proposed solution applies for coding in the transport and application layers.  The proposed approach is in-line with the one in I-D.swett-nwcrg-coding-for-quic.  The proposed solution does not apply for the interaction between coding under the transport layer (i.e. not end-to-end), such as coding for tunnels. </del> <ins> TODO : ADD MORE TEXT </ins> 5. <del> This section presents solutions for a server to add application coding. 5.1. In this solution, the coded packets are sent on top of what is allowed by a congestion window.  Examples of the solution could be adding a given percentage of the congestion window as supplementary packets or sending a given amount of coded packets at a given rate. The redundancy flow can be decorrelated from the congestion control that manages source packets: a secondary congestion control can be introduced, such as in coupled congestion control for RTP media I- D.ietf-rmcat-coupled-cc.  An example would be to exploit a lower than best-effort congestion control RFC6297. The advantage of such a solution is that coding would help in challenging cases where transmission losses are persistent. The drawback of such a solution is that it may result in coding solutions being unfair towards non-coding solutions.  This solutions may result in adding congestion in congested networks. </del> <ins> fig:fec-in illustrates the FEC within the transport case. </ins> <del> 5.2. </del> <ins> fig:fec-in presents an architecture where FEC is within the transport.  The coded packets are sent within what the congestion window allows, such as in CTCP.  Examples of the solution would be sending coded packets when there is no more data to transmit or preferably send coded packets instead of the following packets in the send buffer. </ins> <del> In this solution, the coded packets are sent within what the congestion window allows, such as in CTCP.  Examples of the solution would be sending coded packets when there is no more data to transmit or preferably send coded packets instead of the following packets in the send buffer. The advantage of this solution is that it does not add more traffic than the congestion window allows.  Indeed, all traffic (source and redundancy) is controlled by one congestion control only and TCP metrics for fairness can be indifferently applied in this case. The main drawback is the decrease of goodput if coded packets are sent but are not used at the client side. </del> <ins> TODO : ADD MORE TEXT </ins> 6. <del> [MICHAEL: I would remove this section entirely: it only adds to confusion.  If the inputs are seen as distinct, and we say that the raw inputs shouldn't be shared, then why would one even think about what delay-based CC does with this input?  It's just not the right input.]  Delay-based congestion controls ignore packets that have been repaired with coding.  There is no need to define best current pratices in this case.  However, more discussions are required for congestion controls that use loss as congestion signals (potentially among other congestion detection mechanism). 6.1. [MICHAEL: As above, lets remove all of this please.  This text gets in the way of a clean separation.]  In this solution, the server reacts to repaired packet signals as to congestion-implied packet losses.  That being said, this does not necessarily mean that the packets have actually been lost.  The server may have other means to identify that the packet was just out-of-order and ignore the repaired packet signals. The advantages of the solution are (1) that coding mechanisms do not hide congestion signals, such as packets voluntary dropped by a AQM RFC7567 and (2) packets may be repaired faster than with a traditional retransmission mechanisms. The drawback of this solution is that, if there is a high non- congestion loss rate, the congestion control throughput may decrease drastically.  Reporting this amount of loss to a server may reduce the application goodput when there is no actual congestion. 6.2. [MICHAEL: this is even just a terrible idea.  There's no good rationale to back off less when a packet is repaired.]  In this solution, the server does not reduce the congestion window with the same amount when the "repaired packet" signal is received, i.e. when a packet has been lost but repaired.  Example of this solution could be based on RFC8511 or considering that recovering an isolated packet is not an actual sign of congestion. The advantage of the solution is that in cases where there is no actual congestion, coding could help in improving the transmission without ignoring congestion signals. The main drawback is the precised design of the solution and its interaction with AQM mechanisms RFC7567.  Moreover there may be fairness issues since AIMD convergence may not be guaranteed. 6.3. [MICHAEL: as above, also here, the signals are discussed as going into both blocks, leading to a strange unification of things that shouldn't be together.  Let's remove this.  Also, the idea of ignoring packet loss for CC is appalling anyway.]  This is the case for delay-based congestion controls.  The interaction between delay- based congestion controls and the delay induced by a coding mechanisms is an open research activity.  That being said, a potential approach would be that loss-based congestion control ignores the "repaired packet" signal. The advantage of this solution is that coding would provided substantial benefits in cases where there are transmission losses. However, this solution hides potential congestion losses, making it unfair to other congestion controls. 7. </del> <ins> fig:fec-below illustrates the FEC below the transport case. </ins> <del> [MICHAEL: I'm sorry to say that I find this table to be in the same style, of wrongly mixing things together, and would like to see it removed altogether.]  This section provides a summary on the content in previous sections.  The fig:summary sums up some recommendations. It is worth pointing out that the "coding without congestion" considers that coded packets are sent along with original data packets, in opposition with the solution where coded packets are transmitted only when there is no more original packets to transmit. Moreover, the values indicated in this Figure consider a channel that does not exhibit a high loss pattern. </del> <ins> fig:fec-below presents an architecture where FEC is below the transport.  The coded packets are sent on top of what is allowed by the congestion control.  Examples of the solution could be adding a given percentage of the congestion window as supplementary packets or sending a given amount of coded packets at a given rate.  The redundancy flow can be decorrelated from the congestion control that manages source packets: a secondary congestion control could be introduced underneath the FEC layer, such as in coupled congestion control for RTP media I-D.ietf-rmcat-coupled-cc.  An example would be to exploit a lower than best-effort congestion control RFC6297. </ins> <del> 8. </del> <ins> 7. </ins> This memo includes no request to IANA. <del> 9. </del> <ins> 8. </ins> There is no security considerations required for this document.