Draft: draft-ietf-mptcp-rfc6824bis-11

Commentary: Looks all good !
Paragraph (version A):
This is the same signaling as for initiating a normal TCP connection,
but the SYN, SYN/ACK, and initial ACK (and data) packets also carry
the MP_CAPABLE option.  This option is variable length and serves
multiple purposes.  Firstly, it verifies whether the remote host
supports Multipath TCP; secondly, this option allows the hosts to
exchange some information to authenticate the establishment of
additional subflows.  Further details are given in sec_init.
Paragraph (version B):
This is the same signaling as for initiating a normal TCP connection,
but the SYN, SYN/ACK, and initial ACK (and data) packets also carry
the MP_CAPABLE option.  This option has a variable length and serves
multiple purposes.  Firstly, it verifies whether the remote host
supports Multipath TCP; secondly, this option allows the hosts to
exchange some information to authenticate the establishment of
additional subflows.  Further details are given in sec_init.
Paragraph (version A):
In some circumstances, a host may want to advertise to the remote
host the availability of an address without establishing a new
subflow, for example, when a NAT prevents setup in one direction.  In
the example below, Host A informs Host B about its alternative IP
address/port pair (IP#-A2).  Host B may later send an MP_JOIN to this
new address.  This option contains a HMAC to authenticate the address
as having been sent from the originator of the connection.  The
receiver of this option echoes it back to the client to indicate
successful reception.  Further details are in sec_add_address.
Paragraph (version B):
In some circumstances, a host may want to advertise to the remote
host the availability of an address without establishing a new
subflow, for example, when a NAT prevents setup in one direction.  In
the example below, Host A informs Host B about its alternative IP
address/port pair (IP#-A2).  Host B may later send an MP_JOIN to this
new address.  The ADD_ADDR option contains a HMAC to authenticate the
address as having been sent from the originator of the connection.
The receiver of this option echoes it back to the client to indicate
successful reception.  Further details are in sec_add_address.
Paragraph (version A):
The "Data Sequence Signal" carries the "Data Sequence Mapping".  The
data sequence mapping consists of the subflow sequence number, data
sequence number, and length for which this mapping is valid.  This
option can also carry a connection-level acknowledgment (the "Data
ACK") for the received DSN.
Paragraph (version B):
The Data Sequence Signal (DSS) carries the Data Sequence Mapping.
The Data Sequence Mapping consists of the subflow sequence number,
data sequence number, and length for which this mapping is valid.
This option can also carry a connection-level acknowledgment (the
"Data ACK") for the received DSN.
Paragraph (version A):
To cope with NATs on the path, addresses are referred to by
Address IDs, in case the IP packet's source address gets changed
by a NAT.  Setting up a new TCP flow is not possible if the
passive opener is behind a NAT; to allow subflows to be created
when either end is behind a NAT, MPTCP uses the ADD_ADDR message.
Paragraph (version A):
To meet the threats identified in RFC6181, the following steps are
taken: keys are sent in the clear in the MP_CAPABLE messages;
MP_JOIN messages are secured with HMAC-SHA256 (RFC2104, SHS) using
those keys; and standard TCP validity checks are made on the other
messages (ensuring sequence numbers are in-window RFC5961).
Residual threats to MPTCP v0 RFC6824 were identified in RFC7430,
and those affecting the protocol (i.e. modification to ADD_ADDR)
have been incorporated in this document.  Further discussion of
security can be found in sec_security.
Paragraph (version B):
To cope with NATs on the path, addresses are referred to by
Address IDs, in case the IP packet's source address gets changed
by a NAT.  Setting up a new TCP flow is not possible if the
receiver of the SYN is behind a NAT; to allow subflows to be
created when either end is behind a NAT, MPTCP uses the ADD_ADDR
message.
Paragraph (version B):
To address the threats identified in RFC6181, the following steps
are taken: keys are sent in the clear in the MP_CAPABLE messages;
MP_JOIN messages are secured with HMAC-SHA256 (RFC2104, SHS) using
those keys; and standard TCP validity checks are made on the other
messages (ensuring sequence numbers are in-window RFC5961).
Residual threats to MPTCP v0 RFC6824 were identified in RFC7430,
and those affecting the protocol (i.e. modification to ADD_ADDR)
have been incorporated in this document.  Further discussion of
security can be found in sec_security.
Paragraph (version A):
All MPTCP operations are signaled using optional TCP header fields.
A single TCP option number ("Kind") has been assigned by IANA for
MPTCP (see IANA), and then individual messages will be determined by
a "subtype", the values of which are also stored in an IANA registry
(and are also listed in IANA).
Paragraph (version B):
All MPTCP operations are signaled using optional TCP header fields.
A single TCP option number ("Kind") has been assigned by IANA for
MPTCP (see IANA), and then individual messages will be determined by
a "subtype", the values of which are also stored in an IANA registry
(and are also listed in IANA).  As with all TCP options, the Length
field is specified in bytes, and includes the 2 bytes of Kind and
Length.
Paragraph (version A):
Furthermore, standard TCP validity checks (such as ensuring the
sequence number and acknowledgment number are within window) MUST be
undertaken before processing any MPTCP signals, as described in
RFC5961, and initial subfow sequence numbers SHOULD be generated
according to the recommendations in RFC6528.
Paragraph (version B):
Furthermore, standard TCP validity checks (such as ensuring the
sequence number and acknowledgment number are within window) MUST be
undertaken before processing any MPTCP signals, as described in
RFC5961, and initial subflow sequence numbers SHOULD be generated
according to the recommendations in RFC6528.
Paragraph (version A):
The MP_CAPABLE exchange in this specification (v1) is different to
that specified in v0 RFC6824.  If a host supports multiple versions
of MPTCP, the sender of the MP_CAPABLE option SHOULD signal the
highest version number it supports.  The passive opener, on receipt
of this, will signal the version number it wishes to use, which MUST
be equal to or lower than the version number indicated in the initial
MP_CAPABLE.  There is a caveat though with respect to this version
negotiation with old listeners that only support v0.  A listener that
supports v0 expects that the MP_CAPABLE option in the SYN-segment
includes the initiator's key.  If the initiator however already
upgraded to v1, it won't include the key in the SYN-segment.  Thus,
the listener will ignore the MP_CAPABLE of this SYN-segment and reply
with a SYN/ACK that does not include an MP_CAPABLE, thus leading to a
fallback to regular TCP.  An initiator MAY cache this information
about a peer and for future connections, MAY choose to attempt using
MPTCP v0, if supported, before recording the host as not supporting
MPTCP.
Paragraph (version B):
The MP_CAPABLE exchange in this specification (v1) is different to
that specified in v0 RFC6824.  If a host supports multiple versions
of MPTCP, the sender of the MP_CAPABLE option SHOULD signal the
highest version number it supports.  In return, in its MP_CAPABLE
option, the receiver will signal the version number it wishes to use,
which MUST be equal to or lower than the version number indicated in
the initial MP_CAPABLE.  There is a caveat though with respect to
this version negotiation with old listeners that only support v0.  A
listener that supports v0 expects that the MP_CAPABLE option in the
SYN-segment includes the initiator's key.  If the initiator however
already upgraded to v1, it won't include the key in the SYN-segment.
Thus, the listener will ignore the MP_CAPABLE of this SYN-segment and
reply with a SYN/ACK that does not include an MP_CAPABLE, thus
leading to a fallback to regular TCP.  An initiator MAY cache this
information about a peer and for future connections, MAY choose to
attempt using MPTCP v0, if supported, before recording the host as
not supporting MPTCP.
Paragraph (version A):
The MP_CAPABLE option is carried on the SYN, SYN/ACK, and ACK packets
that start the first subflow of an MPTCP connection, as well as the
first packet that carries data, if the initiator wishs to send first.
The data carried by each option is as follows, where A = initiator
and B = listener.
Paragraph (version B):
The MP_CAPABLE option is carried on the SYN, SYN/ACK, and ACK packets
that start the first subflow of an MPTCP connection, as well as the
first packet that carries data, if the initiator wishes to send
first.  The data carried by each option is as follows, where A =
initiator and B = listener.
Paragraph (version A):
Upon reception of the initial SYN-segment, a stateful server
generates a random key and replies with a SYN/ACK.  The key's method
of generation is implementation specific.  The key MUST be hard to
guess, and it MUST be unique for the sending host at any one time.
Recommendations for generating random numbers for use in keys are
given in RFC4086.  Connections will be indexed at each host by the
token (a one-way hash of the key).  Therefore, an implementation will
require a mapping from each token to the corresponding connection,
and in turn to the keys for the connection.
Paragraph (version A):
There is a risk that two different keys will hash to the same token.
The risk of hash collisions is usually small, unless the host is
handling many tens of thousands of connections.  Therefore, an
implementation SHOULD check its list of connection tokens to ensure
there is not a collision before sending its key, and if there is,
then it should generate a new key.  This would, however, be costly
for a server with thousands of connections.  The subflow handshake
mechanism (sec_join) will ensure that new subflows only join the
correct connection, however, through the cryptographic handshake, as
well as checking the connection tokens in both directions, and
ensuring sequence numbers are in-window.  So in the worst case if
there was a token collision, the new subflow would not succeed, but
the MPTCP connection would continue to provide a regular TCP service.
Paragraph (version B):
Upon reception of the initial SYN-segment, a stateful server
generates a random key and replies with a SYN/ACK.  The key's method
of generation is implementation specific.  The key MUST be hard to
guess, and it MUST be unique for the sending host across all its
current MPTCP connections.  Recommendations for generating random
numbers for use in keys are given in RFC4086.  Connections will be
indexed at each host by the token (a one-way hash of the key).
Therefore, an implementation will require a mapping from each token
to the corresponding connection, and in turn to the keys for the
connection.
Paragraph (version B):
There is a risk that two different keys will hash to the same token.
The risk of hash collisions is usually small, unless the host is
handling many tens of thousands of connections.  Therefore, an
implementation SHOULD check its list of connection tokens to ensure
there is no collision before sending its key, and if there is, then
it should generate a new key.  This would, however, be costly for a
server with thousands of connections.  The subflow handshake
mechanism (sec_join) will ensure that new subflows only join the
correct connection, however, through the cryptographic handshake, as
well as checking the connection tokens in both directions, and
ensuring sequence numbers are in-window.  So in the worst case if
there was a token collision, the new subflow would not succeed, but
the MPTCP connection would continue to provide a regular TCP service.
Paragraph (version A):
delivered.  The receiver will acknowledge this data a the connection
level with a Data ACK, as if a DSS option has been received.
Paragraph (version B):
delivered.  The receiver will acknowledge this data at the connection
level with a Data ACK, as if a DSS option has been received.
Paragraph (version A):
If a SYN contains an MP_CAPABLE option but the SYN/ACK does not, it
is assumed that the passive opener is not multipath capable; thus,
the MPTCP session MUST operate as a regular, single-path TCP.  If a
SYN does not contain a MP_CAPABLE option, the SYN/ACK MUST NOT
contain one in response.  If the third packet (the ACK) does not
contain the MP_CAPABLE option, then the session MUST fall back to
operating as a regular, single-path TCP.  This is to maintain
compatibility with middleboxes on the path that drop some or all TCP
options.  Note that an implementation MAY choose to attempt sending
MPTCP options more than one time before making this decision to
operate as regular TCP (see heuristics).
Paragraph (version B):
If a SYN contains an MP_CAPABLE option but the SYN/ACK does not, it
is assumed that sender of the SYN/ACK is not multipath capable; thus,
the MPTCP session MUST operate as a regular, single-path TCP.  If a
SYN does not contain a MP_CAPABLE option, the SYN/ACK MUST NOT
contain one in response.  If the third packet (the ACK) does not
contain the MP_CAPABLE option, then the session MUST fall back to
operating as a regular, single-path TCP.  This is to maintain
compatibility with middleboxes on the path that drop some or all TCP
options.  Note that an implementation MAY choose to attempt sending
MPTCP options more than one time before making this decision to
operate as regular TCP (see heuristics).
Paragraph (version A):
When receiving a SYN with an MP_JOIN option that contains a valid
token for an existing MPTCP connection, the recipient SHOULD respond
with a SYN/ACK also containing an MP_JOIN option containing a random
number and a truncated (leftmost 64 bits) Hash-based Message
Authentication Code (HMAC).  This version of the option is shown in
tcpm_join2.  If the token is unknown, or the host wants to refuse
subflow establishment (for example, due to a limit on the number of
subflows it will permit), the receiver will send back a reset (RST)
signal, analogous to an unknown port in TCP, containing a MP_TCPRST
option (sec_reset) with an appropriate reason code.  Although
calculating an HMAC requires cryptographic operations, it is believed
that the 32-bit token in the MP_JOIN SYN gives sufficient protection
against blind state exhaustion attacks; therefore, there is no need
to provide mechanisms to allow a responder to operate statelessly at
the MP_JOIN stage.
Paragraph (version B):
When receiving a SYN with an MP_JOIN option that contains a valid
token for an existing MPTCP connection, the recipient SHOULD respond
with a SYN/ACK also containing an MP_JOIN option containing a random
number and a truncated (leftmost 64 bits) Hash-based Message
Authentication Code (HMAC).  This version of the option is shown in
tcpm_join2.  If the token is unknown, or the host wants to refuse
subflow establishment (for example, due to a limit on the number of
subflows it will permit), the receiver will send back a reset (RST)
signal, analogous to an unknown port in TCP, containing a MP_TCPRST
option (sec_reset) with a "MPTCP specific error" reason code.
Although calculating an HMAC requires cryptographic operations, it is
believed that the 32-bit token in the MP_JOIN SYN gives sufficient
protection against blind state exhaustion attacks; therefore, there
is no need to provide mechanisms to allow a responder to operate
statelessly at the MP_JOIN stage.
Paragraph (version A):
If the token received at Host B is unknown or local policy prohibits
the acceptance of the new subflow, the recipient MUST respond with a
TCP RST for the subflow, with a MP_TCPRST option (sec_reset) with an
appropriate reason code.
Paragraph (version B):
If the token received at Host B is unknown or local policy prohibits
the acceptance of the new subflow, the recipient MUST respond with a
TCP RST for the subflow.  If appropriate, a MP_TCPRST option with a
"Administratively prohibited" reason code (sec_reset) should be
included.
Paragraph (version A):
This covers all cases of the loss of an MP_JOIN.  In more detail, if
MP_JOIN is stripped from the SYN on the path from A to B, and Host B
does not have a passive opener on the relevant port, it will respond
with a RST in the normal way.  If in response to a SYN with an
MP_JOIN option, a SYN/ACK is received without the MP_JOIN option
(either since it was stripped on the return path, or it was stripped
on the outgoing path but the passive opener on Host B responded as if
it were a new regular TCP session), then the subflow is unusable and
Host A MUST close it with a RST.
Paragraph (version B):
This covers all cases of the loss of an MP_JOIN.  In more detail, if
MP_JOIN is stripped from the SYN on the path from A to B, and Host B
does not have a listener on the relevant port, it will respond with a
RST in the normal way.  If in response to a SYN with an MP_JOIN
option, a SYN/ACK is received without the MP_JOIN option (either
since it was stripped on the return path, or it was stripped on the
outgoing path but Host B responded as if it were a new regular TCP
session), then the subflow is unusable and Host A MUST close it with
a RST.
Paragraph (version A):
The 'F' flag indicates "DATA_FIN".  If present, this means that this
mapping covers the final data from the sender.  This is the
connection-level equivalent to the FIN flag in single-path TCP.  A
connection is not closed unless there has been a DATA_FIN exchange or
a timeout.  The purpose of the DATA_FIN and the interactions between
this flag, the subflow-level FIN flag, and the data sequence mapping
are described in sec_close.  The remaining reserved bits MUST be set
to zero by an implementation of this specification.
Paragraph (version B):
The 'F' flag indicates "DATA_FIN".  If present, this means that this
mapping covers the final data from the sender.  This is the
connection-level equivalent to the FIN flag in single-path TCP.  A
connection is not closed unless there has been a DATA_FIN exchange,
or an implementation-specific, connection-level timeout.  The purpose
of the DATA_FIN and the interactions between this flag, the subflow-
level FIN flag, and the data sequence mapping are described in
sec_close.  The remaining reserved bits MUST be set to zero by an
implementation of this specification.
Paragraph (version A):
application, and any later data for that sequence space ignored.
Paragraph (version A):
The data sequence number is specified as an absolute value, whereas
the subflow sequence numbering is relative (the SYN at the start of
the subflow has relative subflow sequence number 0).  This is to
allow middleboxes to change the initial sequence number of a subflow,
such as firewalls that undertake ISN randomization.
Paragraph (version B):
application, and any later data for that sequence space should be
ignored.
Paragraph (version B):
The data sequence number is specified as an absolute value, whereas
the subflow sequence numbering is relative (the SYN at the start of
the subflow has relative subflow sequence number 0).  This is to
allow middleboxes to change the initial sequence number of a subflow,
such as firewalls that undertake Initial Sequence Number (ISN)
randomization.
Paragraph (version A):
Once a DATA_FIN has been acknowledged, all remaining subflows MUST be
closed with standard FIN exchanges.  Both hosts SHOULD send FINs on
all subflows, as a courtesy to allow middleboxes to clean up state
even if an individual subflow has failed.  It is also encouraged to
reduce the timeouts (Maximum Segment Life) on subflows at end hosts.
In particular, any subflows where there is still outstanding data
queued (which has been retransmitted on other subflows in order to
get the DATA_FIN acknowledged) MAY be closed with a RST with
MP_TCPRST (sec_reset) error code for "too much outstanding data".
Paragraph (version B):
Once a DATA_FIN has been acknowledged, all remaining subflows MUST be
closed with standard FIN exchanges.  Both hosts SHOULD send FINs on
all subflows, as a courtesy to allow middleboxes to clean up state
even if an individual subflow has failed.  It is also encouraged to
reduce the timeouts (Maximum Segment Lifetime) on subflows at end
hosts after receiving a DATA_FIN.  In particular, any subflows where
there is still outstanding data queued (which has been retransmitted
on other subflows in order to get the DATA_FIN acknowledged) MAY be
closed with a RST with MP_TCPRST (sec_reset) error code for "too much
outstanding data".
Paragraph (version A):
Multiple retransmissions are triggers that will indicate that a
subflow performs badly and could lead to a host resetting the subflow
with a RST.  However, additional research is required to understand
the heuristics of how and when to reset underperforming subflows.
For example, a highly asymmetric path may be misdiagnosed as
underperforming.  A RST for this purpose SHOULD be accompanied with
an appropriate MP_TCPRST option (sec_reset).
Paragraph (version B):
Multiple retransmissions are triggers that will indicate that a
subflow performs badly and could lead to a host resetting the subflow
with a RST.  However, additional research is required to understand
the heuristics of how and when to reset underperforming subflows.
For example, a highly asymmetric path may be misdiagnosed as
underperforming.  A RST for this purpose SHOULD be accompanied with
an "Unacceptable performance" MP_TCPRST option (sec_reset).
Paragraph (version B):
Another use of the MP_PRIO option is to set the 'B' flag on a subflow
to cleanly retire its use before closing it and removing it with
REMOVE_ADDR sec_remove_addr, for example to support make-before-break
session continuity.
Paragraph (version A):
Every address has an Address ID that can be used for uniquely
identifying the address within a connection for address removal.
This is also used to identify MP_JOIN options (see sec_join) relating
to the same address, even when address translators are in use.  The
Address ID MUST uniquely identify the address to the sender (within
the scope of the connection), but the mechanism for allocating such
IDs is implementation specific.
Paragraph (version B):
Every address has an Address ID that can be used for uniquely
identifying the address within a connection for address removal.  The
Address ID is also used to identify MP_JOIN options (see sec_join)
relating to the same address, even when address translators are in
use.  The Address ID MUST uniquely identify the address for the
sender of the option (within the scope of the connection), but the
mechanism for allocating such IDs is implementation specific.
Paragraph (version A):
All address IDs learned via either MP_JOIN or ADD_ADDR SHOULD be
stored by the receiver in a data structure that gathers all the
Address ID to address mappings for a connection (identified by a
token pair).  In this way, there is a stored mapping between Address
ID, observed source address, and token pair for future processing of
control information for a connection.  Note that an implementation
MAY discard incoming address advertisements at will, for example, for
avoiding the required mapping state, or because advertised addresses
are of no use to it (for example, IPv6 addresses when it has IPv4
only).  Therefore, a host MUST treat address advertisements as soft
state, and it MAY choose to refresh advertisements periodically.
Paragraph (version B):
All address IDs learned via either MP_JOIN or ADD_ADDR SHOULD be
stored by the receiver in a data structure that gathers all the
Address ID to address mappings for a connection (identified by a
token pair).  In this way, there is a stored mapping between Address
ID, observed source address, and token pair for future processing of
control information for a connection.  Note that an implementation
MAY discard incoming address advertisements at will, for example, for
avoiding updating mapping state, or because advertised addresses are
of no use to it (for example, IPv6 addresses when it has IPv4 only).
Therefore, a host MUST treat address advertisements as soft state,
and it MAY choose to refresh advertisements periodically.
Paragraph (version A):
Due to the proliferation of NATs, it is reasonably likely that one
host may attempt to advertise private addresses RFC1918.  It is not
desirable to prohibit this, since there may be cases where both hosts
have additional interfaces on the same private network, and a host
MAY want to advertise such addresses.  The MP_JOIN handshake to
create a new subflow (sec_join) provides mechanisms to minimize
security risks.  The MP_JOIN message contains a 32-bit token that
uniquely identifies the connection to the receiving host.  If the
token is unknown, the host will return with a RST.  In the unlikely
event that the token is known, subflow setup will continue, but the
HMAC exchange must occur for authentication.  This will fail, and
will provide sufficient protection against two unconnected hosts
accidentally setting up a new subflow upon the signal of a private
address.  Further security considerations around the issue of
ADD_ADDR messages that accidentally misdirect, or maliciously direct,
new MP_JOIN attempts are discussed in sec_security.
Paragraph (version B):
Due to the proliferation of NATs, it is reasonably likely that one
host may attempt to advertise private addresses RFC1918.  It is not
desirable to prohibit this, since there may be cases where both hosts
have additional interfaces on the same private network, and a host
MAY want to advertise such addresses.  The MP_JOIN handshake to
create a new subflow (sec_join) provides mechanisms to minimize
security risks.  The MP_JOIN message contains a 32-bit token that
uniquely identifies the connection to the receiving host.  If the
token is unknown, the host will return with a RST.  In the unlikely
event that the token is valid at the receiving host, subflow setup
will continue, but the HMAC exchange must occur for authentication.
This will fail, and will provide sufficient protection against two
unconnected hosts accidentally setting up a new subflow upon the
signal of a private address.  Further security considerations around
the issue of ADD_ADDR messages that accidentally misdirect, or
maliciously direct, new MP_JOIN attempts are discussed in
sec_security.
Paragraph (version A):
A host can send an ADD_ADDR message with an already assigned Address
ID, but the Address MUST be the same as previously assigned to this
Address ID, and the Port MUST be different from one already in use
for this Address ID.  If these conditions are not met, the receiver
SHOULD silently ignore the ADD_ADDR.  A host wishing to replace an
existing Address ID MUST first remove the existing one
(sec_remove_addr).
Paragraph (version B):
A host can therefore send an ADD_ADDR message with an already
assigned Address ID, but the Address MUST be the same as previously
assigned to this Address ID.  A new ADD_ADDR may have the same, or
different, port number.  If the port number is different, the
receiving host SHOULD try to set up a new subflow to this new
address/port combination.
Paragraph (version B):
A host wishing to replace an existing Address ID MUST first remove
the existing one (sec_remove_addr).
Paragraph (version A):
If, during the lifetime of an MPTCP connection, a previously
announced address becomes invalid (e.g., if the interface
disappears), the affected host SHOULD announce this so that the peer
can remove subflows related to this address.
Paragraph (version B):
If, during the lifetime of an MPTCP connection, a previously
announced address becomes invalid (e.g., if the interface
disappears), the affected host SHOULD announce this so that the peer
can remove subflows related to this address.  A host MAY also choose
to announce that a valid IP address should not be used any longer,
for example for make-before-break session continuity.
Paragraph (version A):
For security purposes, if a host receives a REMOVE_ADDR option, it
must ensure the affected path(s) are no longer in use before it
instigates closure.  The receipt of REMOVE_ADDR SHOULD first trigger
the sending of a TCP keepalive RFC1122 on the path, and if a response
is received the path SHOULD NOT be removed.  Typical TCP validity
tests on the subflow (e.g., ensuring sequence and ACK numbers are
correct) MUST also be undertaken.  An implementation can use
indications of these test failures as part of intrusion detection or
error logging.
Paragraph (version B):
For security purposes, if a host receives a REMOVE_ADDR option, it
must ensure the affected path(s) are no longer in use before it
instigates closure.  The receipt of REMOVE_ADDR SHOULD first trigger
the sending of a TCP keepalive RFC1122 on the path, and if a response
is received the path SHOULD NOT be removed.  If the path is found to
still be alive, the receiving host SHOULD no longer use the specified
address for future connections, but it is the responsibility of the
host which sent the REMOVE_ADDR to shut down the subflow.  The
requesting host MAY also use MP_PRIO (sec_policy) to request a path
is no longer used, before removal.  Typical TCP validity tests on the
subflow (e.g., ensuring sequence and ACK numbers are correct) MUST
also be undertaken.  An implementation can use indications of these
test failures as part of intrusion detection or error logging.
Paragraph (version A):
Upon receipt of an ACK with MP_FASTCLOSE by Host B, containing the
valid key, Host B answers on the same subflow with a TCP RST and
tears down all subflows.  Host B can now close the whole MPTCP
connection (it transitions directly to CLOSED state).
Paragraph (version B):
Upon receipt of an ACK with MP_FASTCLOSE by Host B, containing the
valid key, Host B answers on the same subflow with a TCP RST and
tears down all subflows also through sending TCP RST signals.
Host B can now close the whole MPTCP connection (it transitions
directly to CLOSED state).
Paragraph (version A):
defined by this specification and are reserved for future use.
Paragraph (version A):
Unspecified error (code 0x0).  This is the default error implying
the subflow is not longer available.  The presence of this option
shows that the RST was generated by a MPTCP-aware device.
Paragraph (version B):
defined by this specification and are reserved for future use.  An
implementation of this specification MUST set these flags to 0, and a
receiver MUST ignore them.
Paragraph (version B):
Unspecified error (code 0x0).  This is the default error implying
the subflow is no longer available.  The presence of this option
shows that the RST was generated by a MPTCP-aware device.
Paragraph (version A):
Sometimes, middleboxes will exist on a path that could prevent the
operation of MPTCP.  MPTCP has been designed in order to cope with
many middlebox modifications (see sec_middleboxes), but there are
still some cases where a subflow could fail to operate within the
MPTCP requirements.  These cases are notably the following: the loss
of MPTCP options on a path and the modification of payload data.  If
such an event occurs, it is necessary to "fall back" to the previous,
safe operation.  This may be either falling back to regular TCP or
removing a problematic subflow.
Paragraph (version B):
Sometimes, middleboxes will exist on a path that could prevent the
operation of MPTCP.  MPTCP has been designed in order to cope with
many middlebox modifications (see sec_middleboxes), but there are
still some cases where a subflow could fail to operate within the
MPTCP requirements.  These cases are notably the following: the loss
of MPTCP options on a path, and the modification of payload data.  If
such an event occurs, it is necessary to "fall back" to the previous,
safe operation.  This may be either falling back to regular TCP or
removing a problematic subflow.
Paragraph (version A):
If, however, an ACK is received for data (not just for the SYN)
without a DSS option containing a Data ACK, the sender determines the
path is not MPTCP capable.  In the case of this occurring on an
additional subflow (i.e., one started with MP_JOIN), the host MUST
close the subflow with a RST.  In the case of the first subflow
(i.e., that started with MP_CAPABLE), it MUST drop out of an MPTCP
mode back to regular TCP.  The sender will send one final data
sequence mapping, with the Data-Level Length value of 0 indicating an
infinite mapping (in case the path drops options in one direction
only), and then revert to sending data on the single subflow without
any MPTCP options.
Paragraph (version B):
If, however, an ACK is received for data (not just for the SYN)
without a DSS option containing a Data ACK, the sender determines the
path is not MPTCP capable.  In the case of this occurring on an
additional subflow (i.e., one started with MP_JOIN), the host MUST
close the subflow with a RST, which SHOULD contain a MP_TCPRST option
(sec_reset) with a "Middlebox interferance" reason code.
Paragraph (version B):
In the case of such an ACK being received on the first subflow (i.e.,
that started with MP_CAPABLE), before any additional subflows are
added, the implementation MUST drop out of an MPTCP mode, back to
regular TCP.  The sender will send one final data sequence mapping,
with the Data-Level Length value of 0 indicating an infinite mapping
(to inform the other end in case the path drops options in one
direction only), and then revert to sending data on the single
subflow without any MPTCP options.
Paragraph (version A):
If a subflow breaks during operation, e.g. if it is re-routed and
MPTCP options are no longer permitted, then once this is detected (by
the subflow-level receive buffer filling up), the subflow SHOULD be
treated as broken and closed with a RST, since no data can be
delivered to the application layer, and no fallback signal can be
reliably sent.  This RST SHOULD include the MP_TCPRST option
(sec_reset) with an appropriate reason code.
Paragraph (version B):
If a subflow breaks during operation, e.g. if it is re-routed and
MPTCP options are no longer permitted, then once this is detected (by
the subflow-level receive buffer filling up), the subflow SHOULD be
treated as broken and closed with a RST, since no data can be
delivered to the application layer, and no fallback signal can be
reliably sent.  This RST SHOULD include the MP_TCPRST option
(sec_reset) with a "Middlebox interferance" reason code.
Paragraph (version A):
These rules should cover all cases where such a failure could happen:
whether it's on the forward or reverse path and whether the server or
the client first sends data.  If lost options on data packets occur
on any other subflow apart from the initial subflow, it should be
treated as a standard path failure.  The data would not be DATA_ACKed
(since there is no mapping for the data), and the subflow can be
closed with a RST, containing a MP_TCPRST option (sec_reset) with an
appropriate reason code.
Paragraph (version A):
The case described above is a specialized case of fallback, for when
the lack of MPTCP support is detected before any data is acknowledged
at the connection level on a subflow.  More generally, fallback
(either closing a subflow, or to regular TCP) can become necessary at
any point during a connection if a non-MPTCP-aware middlebox changes
the data stream.
Paragraph (version A):
As described in sec_generalop, each portion of data for which there
is a mapping is protected by a checksum, if checksums have been
negotiated.  This mechanism is used to detect if middleboxes have
made any adjustments to the payload (added, removed, or changed
data).  A checksum will fail if the data has been changed in any way.
This will also detect if the length of data on the subflow is
increased or decreased, and this means the data sequence mapping is
no longer valid.  The sender no longer knows what subflow-level
sequence number the receiver is genuinely operating at (the middlebox
will be faking ACKs in return), and it cannot signal any further
mappings.  Furthermore, in addition to the possibility of payload
modifications that are valid at the application layer, there is the
possibility that false positives could be hit across MPTCP segment
boundaries, corrupting the data.  Therefore, all data from the start
of the segment that failed the checksum onwards is not trustworthy.
Paragraph (version B):
These rules should cover all cases where such a failure could happen:
whether it's on the forward or reverse path and whether the server or
the client first sends data.  If lost options on data packets occur
on any other subflow apart from the initial subflow, it should be
treated as a standard path failure.  The data would not be DATA_ACKed
(since there is no mapping for the data), and the subflow can be
closed with a RST, containing a MP_TCPRST option (sec_reset) with a
"Middlebox interferance" reason code.
Paragraph (version B):
So far this section has discussed the lost of MPTCP options, either
initially, or during the course of the connection.  As described in
sec_generalop, each portion of data for which there is a mapping is
protected by a checksum, if checksums have been negotiated.  This
mechanism is used to detect if middleboxes have made any adjustments
to the payload (added, removed, or changed data).  A checksum will
fail if the data has been changed in any way.  This will also detect
if the length of data on the subflow is increased or decreased, and
this means the data sequence mapping is no longer valid.  The sender
no longer knows what subflow-level sequence number the receiver is
genuinely operating at (the middlebox will be faking ACKs in return),
and it cannot signal any further mappings.  Furthermore, in addition
to the possibility of payload modifications that are valid at the
application layer, there is the possibility that such modifications
could be triggered across MPTCP segment boundaries, corrupting the
data.  Therefore, all data from the start of the segment that failed
the checksum onwards is not trustworthy.
Paragraph (version A):
When multiple subflows are in use, the data in flight on a subflow
will likely involve data that is not contiguously part of the
connection-level stream, since segments will be spread across the
multiple subflows.  Due to the problems identified above, it is not
possible to determine what the adjustment has done to the data
(notably, any changes to the subflow sequence numbering).  Therefore,
it is not possible to recover the subflow, and the affected subflow
must be immediately closed with a RST, featuring an MP_FAIL option
(tcpm_fallback), which defines the data sequence number at the start
of the segment (defined by the data sequence mapping) that had the
checksum failure.  Note that the MP_FAIL option requires the use of
the full 64-bit sequence number, even if 32-bit sequence numbers are
normally in use in the DSS signals on the path.
Paragraph (version A):
The receiver MUST discard all data following the data sequence number
specified.  Failed data MUST NOT be DATA_ACKed and so will be
retransmitted on other subflows (sec_retransmit).
Paragraph (version B):
When multiple subflows are in use, the data in flight on a subflow
will likely involve data that is not contiguously part of the
connection-level stream, since segments will be spread across the
multiple subflows.  Due to the problems identified above, it is not
possible to determine what adjustment has done to the data (notably,
any changes to the subflow sequence numbering).  Therefore, it is not
possible to recover the subflow, and the affected subflow must be
immediately closed with a RST, featuring an MP_FAIL option
(tcpm_fallback), which defines the data sequence number at the start
of the segment (defined by the data sequence mapping) that had the
checksum failure.  Note that the MP_FAIL option requires the use of
the full 64-bit sequence number, even if 32-bit sequence numbers are
normally in use in the DSS signals on the path.
Paragraph (version B):
The receiver of this option MUST discard all data following the data
sequence number specified.  Failed data MUST NOT be DATA_ACKed and so
will be retransmitted on other subflows (sec_retransmit).
Paragraph (version A):
stream is equivalent to a regular, legacy TCP session.  The MP_FAIL
signal affects only one direction of traffic.  It is not mandatory
for the receiver of an MP_FAIL to also respond with an MP_FAIL, since
the paths may only be damaged in one direction.  However,
implementations MAY choose to send a MP_FAIL in the reverse direction
and entirely revert to a regular TCP session.
Paragraph (version B):
stream is equivalent to a regular, legacy TCP session.  Whilst in
theory paths may only be damaged in one direction, and the MP_FAIL
signal affects only one direction of traffic, for implementation
simplicity, the receiver of an MP_FAIL MUST also respond with an
MP_FAIL in the reverse direction and entirely revert to a regular TCP
session.
Paragraph (version A):
When a connection has fallen back, only one subflow can send data;
otherwise, the receiver would not know how to reorder the data.  In
practice, this means that all MPTCP subflows will have to be
terminated except one.  Once MPTCP falls back to regular TCP, it MUST
NOT revert to MPTCP later in the connection.
Paragraph (version A):
It should be emphasized that we are not attempting to prevent the use
of middleboxes that want to adjust the payload.  An MPTCP-aware
middlebox could provide such functionality by also rewriting
checksums.
Paragraph (version B):
When a connection has fallen back with an infinite mapping, only one
subflow can send data; otherwise, the receiver would not know how to
reorder the data.  In practice, this means that all MPTCP subflows
will have to be terminated except one.  Once MPTCP falls back to
regular TCP, it MUST NOT revert to MPTCP later in the connection.
Paragraph (version B):
It should be emphasized that MPTCP is not attempting to prevent the
use of middleboxes that want to adjust the payload.  An MPTCP-aware
middlebox could provide such functionality by also rewriting
checksums.
Paragraph (version A):
There may also be cases, however, where the passive opener wishes to
signal to the other host that a specific port should be used, and
this facility is provided in the Add Address option as documented in
sec_add_address.  It is therefore feasible to allow multiple subflows
between the same two addresses but using different port pairs, and
such a facility could be used to allow load balancing within the
network based on 5-tuples (e.g., some ECMP implementations RFC2992).
Paragraph (version B):
There may also be cases, however, where a host wishes to signal that
a specific port should be used, and this facility is provided in the
ADD_ADDR option as documented in sec_add_address.  It is therefore
feasible to allow multiple subflows between the same two addresses
but using different port pairs, and such a facility could be used to
allow load balancing within the network based on 5-tuples (e.g., some
ECMP implementations RFC2992).
Paragraph (version A):
Note that this heuristic alone is probably insufficient.  Traffic for
many common applications, such as downloads, is highly asymmetric and
the host that is multihomed may well be the client that will never
fill its buffers, and thus never use MPTCP.  Advanced APIs that allow
an application to signal its traffic requirements would aid in these
decisions.
Paragraph (version B):
Note that this heuristic alone is probably insufficient.  Traffic for
many common applications, such as downloads, is highly asymmetric and
the host that is multihomed may well be the client that will never
fill its buffers, and thus never use MPTCP according to this
heuristic.  Advanced APIs that allow an application to signal its
traffic requirements would aid in these decisions.
Paragraph (version A):
The (in-header) TCP sequence number is specific to the subflow.
To allow the receiver to reorder application data, an additional
data-level sequence space is used.  In this data-level sequence
space, the initial SYN and the final DATA_FIN occupy 1 octet of
sequence space.  There is an explicit mapping of data sequence
space to subflow sequence space, which is signaled through TCP
options in data packets.
Paragraph (version B):
The (in-header) TCP sequence number is specific to the subflow.
To allow the receiver to reorder application data, an additional
data-level sequence space is used.  In this data-level sequence
space, the initial SYN and the final DATA_FIN occupy 1 octet of
sequence space.  This is to ensure these signals are acknowledged
at the connection level.  There is an explicit mapping of data
sequence space to subflow sequence space, which is signaled
through TCP options in data packets.
Paragraph (version A):
The case when options get stripped on data packets has been discussed
in the Fallback section.  If a fraction of options are stripped,
Paragraph (version B):
The case when options get stripped on data packets has been discussed
in the Fallback section.  If only some MPTCP options are stripped,
Paragraph (version A):
NATs RFC3022 (Network Address (and Port) Translators) change the
source address (and often source port) of packets.  This means
that a host will not know its public-facing address for signaling
in MPTCP.  Therefore, MPTCP permits implicit address addition via
the MP_JOIN option, and the handshake mechanism ensures that
connection attempts to private addresses RFC1918 do not cause
problems.  Explicit address removal is undertaken by an Address ID
to allow no knowledge of the source address.
Paragraph (version B):
NATs RFC3022 (Network Address (and Port) Translators) change the
source address (and often source port) of packets.  This means
that a host will not know its public-facing address for signaling
in MPTCP.  Therefore, MPTCP permits implicit address addition via
the MP_JOIN option, and the handshake mechanism ensures that
connection attempts to private addresses RFC1918, since they are
authenticated, will only set up subflows to the correct hosts.
Explicit address removal is undertaken by an Address ID to allow
no knowledge of the source address.
Paragraph (version A):
Intrusion Detection Systems look out for traffic patterns and
content that could threaten a network.  Multipath will mean that
such data is potentially spread, so it is more difficult for an
IDS to analyze the whole traffic, and potentially increases the
risk of false positives.  However, for an MPTCP-aware IDS, tokens
can be read by such systems to correlate multiple subflows and
reassemble for analysis.
Paragraph (version B):
Intrusion Detection Systems look out for traffic patterns and
content that could threaten a network.  Multipath will mean that
such data is potentially spread, so it is more difficult for an
IDS to analyze the whole traffic, and potentially increases the
risk of false positives.  However, a MPTCP-aware IDS can read
tokens to correlate multiple subflows and reassemble them for
analysis.
Paragraph (version A):
The authors also wish to acknowledge reviews and contributions from
Iljitsch van Beijnum, Lars Eggert, Marcelo Bagnulo, Robert Hancock,
Pasi Sarolahti, Toby Moncaster, Philip Eardley, Sergio Lembo,
Lawrence Conroy, Yoshifumi Nishida, Bob Briscoe, Stein Gjessing,
Andrew McGregor, Georg Hampel, Anumita Biswas, Wes Eddy, Alexey
Melnikov, Francis Dupont, Adrian Farrel, Barry Leiba, Robert Sparks,
Sean Turner, Stephen Farrell, Martin Stiemerling, Gregory Detal, and
Fabien Duchene.
Paragraph (version B):
The authors also wish to acknowledge reviews and contributions from
Iljitsch van Beijnum, Lars Eggert, Marcelo Bagnulo, Robert Hancock,
Pasi Sarolahti, Toby Moncaster, Philip Eardley, Sergio Lembo,
Lawrence Conroy, Yoshifumi Nishida, Bob Briscoe, Stein Gjessing,
Andrew McGregor, Georg Hampel, Anumita Biswas, Wes Eddy, Alexey
Melnikov, Francis Dupont, Adrian Farrel, Barry Leiba, Robert Sparks,
Sean Turner, Stephen Farrell, Martin Stiemerling, Gregory Detal,
Fabien Duchene, Xavier de Foy, and Rahul Jadhav.