Draft: draft-irtf-nwcrg-coding-and-congestion-04

Commentary: My 2 cents and it ’s probably not worth more : I am following the evolution of this draft and while the issues are starting to be well covered this is RG document so while I approve of the changes it would be nice to highlight the research that is still needed .      , Ph.D. 
 Regarding     comment : I agree , and I think that we could approach this by trying to find at least one published example of research per category - ideally the " best " per category . This would show , wherever we find something , what the SoA is - research could focus on improving this SoA. Also , it would probably give us some holes - e.g. , I do n't think there 's any much research on ( e2e ! ) FEC " below the transport " , especially using multiple CC instances as hinted on in the draft . These holes would let us identify missing research . 
 You should define what you mean by fairness . I remember endless discussions about fairness in NC with   amongst others .     , Ph.D. 
 I have corrected most of the comments and issues . I have two issues to discuss the discussion points opened by    . How about we merge this and have a new fresh start ? ( from which it will be easier for you to create branches , etc . ) I guess , looking at the amount of modifications , : may you merge if OK with you ? 
 Ah sorry , I see the comment now , asking me to merge - ok , then I click these 3 " commit"s and then I merge ! Doing it now ! 
  made a comment on partial reliability . Should we need a new subsection 6.X to discuss this ? 
 I do n't get this :  's comment was about partial order , not partial reliability . Quote : One important thing to consider is reordering and not only reliability alone , we could also consider Partial Order Connection ( POC ) and Partial Order Transport Service ( see   works ) . Spurious retransmission is not only linked to reliability but reordering . 
 If this issue ends up being about ordering , and not partial reliability : From  ( copied over from a PR ): There is something we do not mention here concerning the reliable transport that deliver data in - order : with a reliable transport , the data are delivered in order . I can see two possible cases : Above a single TCP connection : the repair packets wo n't ever be delivered to the applications if there is a loss occurring before them . They are thus useless as they are delivered once the loss is receovered . Above multiple streams ( e.g. QUIC streams or one TCP connection for the source and one for the repair symbols ): the repair symbols are less effective : if there is a hole in the data ( ex : seq 1 and 3 are received but not 2),you will need to consider 3 a lost too as it wo n't be delivered to the upper layer . 
 An answer from me to this : there are transports that can deliver data out - of - order . SCTP can do that , and it 's exposed in the interface of  ( draft - ietf - taps - interface ) . Multi - streaming is closely related to that : if you take out - of - order message delivery and just add stream identifiers to ensure that , per stream , messages are delivered in order , you have it . This means that FEC could perhaps be applied across streams in case of a multi - streaming transport . I think this just complicates the discussion , let 's not go there in the draft . 
 Thank you for summing up the discussion here . I think we already quite an important number of new sections in this document - we may leave the topic on out - of - order packets on the side and focus on what we have at the moment . 
 Ok , but do we agree that this is about " partial ordering " , not " partial reliability " ? In this case , we should probably rename this issue to avoid further confusion . 
 Text proposed in PR 
 Should we consider the impact of cross trafic that uses congestion - controled transport protocol ? 
 done in PR 
 There should be a discussion on the amount of redundancy of the FEC e.g. FEC above transport : including redundancy reduces goodput for large files transfers e.g. FEC in transport : including redundancy reduces goodput for large files transfers but can be adapted depending on the CWND size e.g. FEC below transport : including redundancy adds congestions without reducing goodput 
 From   ( IETF108 ): One thing it might be good to cover is how to couple increasing transmission rate with increasing / decreasgin coding rate . It may be interesting to use increasing coding rate as a " probe " to more safely increase transmission rate . Has this been looked at ? 
 Done in PR 
 More related work , examples of FEC research work ? Papers ? This kind of thing ? ( I agree , just clarifying if this is what this issue is asking for ) 
 Started in PR 
 The title of the document is " Coding and congestion control in transport " but we only present a " FEC Channel " that only carries repair packets and some signaling about the repaired packets . I like the idea of two separated channels a lot , but I think we also should integrate source packets as part of the FEC channel , and maybe rename the FEC channel to " Coding Channel " and " packets " to " symbols " when we talk about the coding channel ? Here is what I propose concretely : The CC channel stays unchanged , it carries packets The FEC channel becomes a Coding channel and carries symbols : Source Symbols and Repair Symbols . I think the approach there is more general : one network packet can contain several Source / Repair Symbols ( similarly to the coding - for - quic draft ) . I think this helps to isolate both CC and FEC channels from each others as the FEC / Coding channel only plays with Symbols while the CC channel plays with packets . When I read the current design , it makes me think that the FEC channel needs the source packets from the CC channel to decode the received repair packets but this is shown nowhere in the graph . By considering different entities ( packets VS symbols that are in fact a view on the sent / received data ) , we decorrelate more intuitively the FEC and the CC channels . As a consequence , we will only talk about symbols when it concerns the Coding channel and packets when it concerns the CC channel . What do you think ? I do not think this is an urging change by the way . 
 Hi , Coding channel refers to physical layer and when combined with the term symbol , it also refers to physical operations which are not those at packet level . We stated in RFC8406 that Generally speaking , a Packet is the unit of data that is sent in the Packet Erasure Channel , while a Symbol is the unit of data that is manipulated during the encoding and decoding operations . So we allow the use of symbol only when it corresponds to non yet transmitted data . As soon as transmitted , it becomes a packet . EL 
 Hello , symbol , it also refers to physical operations which are not those at packet level .  , in that case , renaming the channel this way would make it more confusing I agree , sorry for that then :-) transmitted data . As soon as transmitted , it becomes a packet . I think the definition I proposed here still fits with the symbol definition you described though . A packet can contain more than 1 source / repair symbol(s ) and the source symbols can be different than what is actually sent on the wire ( e.g. source symbols can contain padding that is not actually sent on the wire or can contain metadata that are not sent on the wire either ) . So I still think that it matches more the definition of symbols than packets . All the real packets received by a protocol can be processed and transformed into source symbols before being passed to the FEC Channel . ( sorry in advance if I have misconceptions on this matter ) 
 Yes in that context it 's fine and we stick to RFC8406 . 
 Done in PR 
 DO : Fine , but you miss the opposite case : how many repair packets are you sending that turn out to be useless ? It 's a matter of measuring the usefulness of repair traffic . It gives an extra degree of freedom as you may want to play at the congestion control level or at the coding rate level . NK : Good point , that 's the kind of topic we need to gather in the document . There are several corner cases that are very important . This point is about how you add coding inside congestion control . With this claim , we want to be generic . 
 I think this document ought to remove the confusion between coding and congestion control that folks seem to have . So : " It gives an extra degree of freedom as you may want to play at the congestion control level or at the coding rate level . " sounds like it 's based on this confusion . And : " This point is about how you add coding inside congestion control . " does n't seem right either . 
 Indeed - the introduction of the coding and congestion channels could help . I may need to be reflected in the rest of the document . 
 Maybe we could emphasize on the fact that coding is a reliability mechanism and is distinct and separated from the loss detection and congestion control mechanisms . If all we care about is bandwidth in a general unicast scenario , sending a - priori repair packets should not help more than Selective - Repeat ARQ . In the best case , it should perform as well as a SR - ARQ if the packet is lost ( without additional gain ) , and will simply waste bandwidth compared to SR - ARQ if the packet is not lost . I can identify one case where doing FEC can help for bandwidth : when the buffer are full or very limited and can not be in - order delivered to the application because of head - of - line blocking . Sending FEC could help to unblock the peer 's receive buffer sooner . But this has nothing to do with congestion control . 
 I think this kind of discussion should be in the document . 
 I agree with 's idea of unblocking the receiver . Also , in the other issue , he mentioned the big disadvantage ( delay ) coming from tail losses . These things should now be to the document - now it 's structured such that it could be better , in the right place . Other thoughts that were brought up in discussions , and should go into the doc - both are about the FEC - below - transport case : 1 ) ECN : From : There is more detail about possibilities to use ECN for FEC - below - transport in Emmanuel 's presentation : ( slide 4 ) 2 ) special environments : and we had this discussion in a PR related to FEC - below - transport : wrote : and I answered : .... and about the FEC - above - transport case slide 2 reminds me that this is obviously useless if the transport repairs all losses , as with TCP ! This is worth mentioning in the FEC - above - transport case - we could mention that unreliable congestion - controlled transports ( e.g.  cc 's , or DCCP ) are better to consider here than fully reliable ones such as TCP . 
 I am unsure where this issue is going and what we can integrate in the draft then . I would say that identifying useless repair packets is the work of the reliability mechanism ( there is a clear similarity with spurious retransmissions detection ) and in my opinion has nothing to do with congestion control , so what can we integrate in the draft ? 
 I think this discussion is covered in PR 
 Thanks for the PR ! : -)There are two small suggestions that I made ... not sure they 're committed yet ? Else just press " commit " 3 times please :)
fig:sep-channel presents the notations that will be used in this
document and introduces the Congestion Control (CC) and Forward
Erasure Correction (FEC) channels.  The Congestion Control channel
carries source packets from a sender to a receiver, and packets
signaling information about the network (number of packets received
vs. lost, ECN marks, etc.) from the receiver to the sender.  The
Forward Erasure Correction channel carries repair packets (from the
sender to the receiver) and potential information signaling which
packets have been repaired (from the receiver to the sender).  It is
worth pointing out that there are cases where these channels are not
separated.
fig:sep-channel presents the notations that will be used in this
document and introduces the Congestion Control (CC) and Forward
Erasure Correction (FEC) channels.  The Congestion Control channel
carries source packets from a sender to a receiver, and packets
signaling information about the network (number of packets received
vs. lost, ECN marks, etc.) from the receiver to the sender.  The
Forward Erasure Correction channel carries repair symbols (from the
sender to the receiver) and potential information signaling which
packets have been repaired (from the receiver to the sender).  It is
worth pointing out that there are cases where these channels are not
separated.
'signaling repaired packets' (input control plane for the FEC):
refers to the information a FEC sender can obtain from a FEC
receiver about the performance of the FEC solution as seen from
the receiver.
'signaling repaired symbols' (input control plane for the FEC):
refers to the information a FEC sender can obtain from a FEC
receiver about the performance of the FEC solution as seen from
the receiver.
'source and/or repair packets' (data plane for both the FEC and
the CC): refers to the packets that are transmitted.  There can
only be source packets (if the coding rate is 0), repair packets
(if the solution decides not to send the original source packets)
or a mix of both.
The inputs to FEC (packets to work upon, and signaling from the
receiver about losses and/or repaired packets) are distinct from the
inputs to CC.  The latter calculates a sending rate or window from
network information, and it takes the packet to send as input,
sometimes along with application requirements such as upper/lower
rate bounds, periods of quiescence, or a priority.  It is not clear
that the ACK signals feeding into a congestion control algorithm are
useful to FEC in their raw form, and vice versa - information about
repaired blocks may be quite irrelevant to a CC algorithm.
'source and/or repair symbols' (data plane for both the FEC and
the CC): refers to the data that is transmitted.  The sender can
decide to send source symbols only (meaning that the coding rate
is 0), repair symbols only (if the solution decides not to send
the original source packets) or a mix of both.
The inputs to FEC (packets to work upon, and signaling from the
receiver about losses and/or repaired symbols) are distinct from the
inputs to CC.  The latter calculates a sending rate or window from
network information, and it takes the packet to send as input,
sometimes along with application requirements such as upper/lower
rate bounds, periods of quiescence, or a priority.  It is not clear
that the ACK signals feeding into a congestion control algorithm are
useful to FEC in their raw form, and vice versa - information about
repaired blocks may be quite irrelevant to a CC algorithm.
While CC is in principle independent of other transport functions
such as reliability, we note that CC is often embedded in reliable
transfer protocols (e.g.  TCP).  This approach may require that the
transport protocol does not implement a fully reliable data transfer
service (e.g., based on lost packet retransmission), otherwise this
could lead to spurious retransmissions.  UDP is an example of a
protocol for which this approach is relevant.
The advantage of this approach is that the FEC does not contribute to
adding congestion in the network.  When a congestion control is
implementedd at the transport layer, the repair packets are sent
within what the congestion window allows.
fig:fec-in presents an architecture where FEC is within the
transport.  The repair packets are sent within what the congestion
window allows, such as in CTCP.
The advantage of this approach is that it can enable conjoint
optimization between the CC and the FEC.  Moreover, the transmission
of repair packets does not add congestion in potentially congested
networks but helps repair lost packets (such as tail losses).
The drawback of this approach is that it may not result in much gains
as opposed to classical retransmission mechanisms and it costs
bandwidth that could have been exploited to transmit source packets.
The coding ratio needs to be carefully designed.
Examples of the solution would be sending repair packets when there
is no more data to transmit or preferably send repair packets instead
of the following packets in the send buffer.
fig:fec-below presents an architecture where FEC is below the
transport.  The repair packets are sent on top of what is allowed by
the congestion control.
The advantage of this approach is that the FEC overhead does not
contribute to congestion in the network.  When congestion control is
implemented at the transport layer, the repair symbols are sent
within what the congestion window allows.  This approach can result
in improved quality of experience for latency sensitive and non
reliable applications.
This approach requires that the transport protocol does not implement
a reliable data transfer service (e.g., based on lost packet
retransmission).  UDP is an example of a protocol for which this
approach is relevant.  For reliable transfers, the usage of coding
does not guarantee better performance and would mainly reduce goodput
for large files transfers.
fig:fec-in presents an architecture where FEC is within the
transport.  The repair symbols are sent within what the congestion
window allows, such as in CTCP.
The advantage of this approach is that it can enable conjoint
optimization between the CC and the FEC.  Moreover, the transmission
of repair symbols does not add congestion in potentially congested
networks but helps repair lost packets (such as tail losses).
For reliable transfers, including redundancy reduces goodput for
large files transfers but the amount of repaired symbols can be
adapted, e.g. depending on the congestion window size.  There is a
trade-off between the cost in bandwidth that could have been
exploited to transmit source packets and the advantages that are
brought by transmitting repair symbols (e.g. unlocking the receive
buffer if this is limiting).  The coding ratio needs to be carefully
designed.  For small files, sending repair symbols when there is no
more data to transmit or preferably sending repair symbols instead of
the following packets in the send buffer could help to reduce the
transfer time.  This is because it could avoid a silent period
between the transmission of the last packet and firing the
retransmission of lost packets.
fig:fec-below presents an architecture where FEC is applied end-to-
end below the transport layer, but above the link layer.  Note that
it is common to apply FEC at the link layer, in which it contributes
to the total capacity that a link exposes to upper layers.  This
application of FEC is out of scope of this document.  The repair
symbols are sent on top of what is allowed by the congestion control.
In this case, including redundancy adds congestion without reducing
goodput but leads to potential fairness issues.  Indeed, the
effective bitrate is higher than the CC's computed fair share due to
the sending of repair symbols and the losses are hidden to the
transport which may cause a problem for loss-based congestion
controls, but is not a problem for delay-based ones.
Examples of the solution could be adding a given percentage of the
congestion window as supplementary packets or sending a given amount
of repair packets at a given rate.  The redundancy flow can be
decorrelated from the congestion control that manages source packets:
a secondary congestion control could be introduced underneath the FEC
layer.  The separate congestion control mechanisms could be made to
work together while adhering to priorities, as in coupled congestion
control for RTP media I-D.ietf-rmcat-coupled-cc.  Another possibility
would be to exploit a lower than best-effort congestion control
RFC6297 for repair packets.
Examples of the solution could be adding a given percentage of the
congestion window as supplementary symbols or sending a given amount
of repair symbols at a given rate.  The redundancy flow can be
decorrelated from the congestion control that manages source packets:
a separate congestion control could be introduced to manage the
amount of repaired packets to transmit on the FEC channel.  The
separate congestion control instances could be made to work together
while adhering to priorities, as in coupled congestion control for
RTP media RFC8699 in case all traffic can be assumed to take the same
path, or otherwise with a multipath congestion window coupling
mechanism as in Multipath TCP RFC6356.  Another possibility would be
to exploit a lower than best-effort congestion control RFC6297 for
repair symbols.
The objective of this section is to further detail some aspects that
have been expressed in previous discussion subsections.
The objective of this section is to further discuss how adding coded
flows impacts on non-coded flows when they share the same bottleneck.
This section assumes that the non-coded flows respond to congestion
signals from the network.
The addition of coding within the flow do not impact on how the coded
and non-coded flows interact.  This interaction would mainly depend
on the congestion controls embedded in each host.
The addition of coding within the flow may impact on the congestion
control mechanism and hide congestion losses.  Specific interaction
between congestion controls and coding schemes can be proposed (see
subsec:cc-recov-interaction, subsec:cc-nc-interaction and subsec:cc-
useless-interaction).  If no specific interaction is introduced, the
coding scheme may hide congestion losses to the congestion controller
and the description of subsub:fec_below may apply.
In this case, the coding scheme may hide congestion losses to the
congestion controller.  There are cases where this can drastically
reduce the goodput of the non-coded flows.  Depending on the
congestion control, it may be possible to indicate that a packet has
been recovered, e.g. using ECN flags, to reduce the impact on the
non-coded flows (see subsec:cc-recov-inter-below).
The objective of this subsection is to destribe potential
interactions between the congestion control and the recovered
symbols.
The congestion control may not be able to differentiate repair
symbols from actual source packets.  The relevance of adding coding
at the application layer is related to the needs of the application.
For real-time applications, this approach may reduce the amount of
retransmission.  The usage of a non-reliable transport is more
adequate in this case.
If the two FEC and CC channels are decoupled, the endpoint may
exploit specific protocol for each of the channel.  The channels may
be coupled and one single protocol may be exploited.  In both cases,
the receiver can differenciate source packets and repair symbols.
The receiver may indicate the amount of source packets that have been
received and the amount of repaired symbols that were actually useful
in the recovery process of packets.
QUIC may require a specific type of frame to transport such
information between receiver and sender.
The congestion control may not know what is going on in the network
underneath and whether a coding scheme is introduced or not.  The
congestion control may behave as if no coding sheme is introduced.
The only way for a coding channel to indicate that symbols have been
recovered is to exploit existing signaling.  An example would be a
receiver to indicate that a packet has been recovered by using ECN
signals of TCP.
This section discuss to what extent interaction between the
congestion control and the coding rates are possible.
The coding rate applied at the application layer mainly depends on
the available capacity given by the congestion control underneath.
Adapting the coding rate to the minimum required data rate of the
application may help to reduce packet losses and improve quality of
experience.
In this case, there is an important flexibility in the trade-off
between reducing goodput, helping recovering from transmission and
congestion losses, being fair in respect with non-coded flows.  Such
as explained in subsec:cc-recov-inter-in, the receiver may be able to
indicate the sender the amount of packets that have been received or
recovered.  The sender may exploit this information to tune the
coding ratio.  As one example of flexibility of this case, coupling
an increasing transmission rate with an increasing or decreasing
coding rate could be envisioned.  A server may use increasing coding
rate as a probe of the channel capacity and adapt the transmission
rate of the congestion control.
The coding rate in this case can be tuned depending on the amount of
recovered symbols and the rate of which data are transmitted by the
sender.  The coding scheme is not aware of the congestion control
implementation, making it hard for the coding scheme to apply the
relevant coding rate.
There are cases where useless repaired symbols may be transmitted.
These have an impact on the network load and may reduce the goodput
of the flow without concrete gains.
In this case, the discussion depends on application needs.  The only
case where adding useless repaired symbols does not result in reduced
goodput is when the application needs a limited amount of goodput
(e.g.  VoIP traffic).  In this case, the useless repaired symbols
would only have an impact on the amount of data generated in the
network.
The sender may be able to exploit the information given by the
receiver to reduce the amount of useless repair packets and the
resulting goodput reduction.
In this case, the useless repaired symbols only have an impact on the
load of the network without actual gain for the coded flow.