Draft: draft-brunstrom-taps-impl

Commentary: In the Implementation draft , we add text to explain how Application Intents can be implemented in a transport system . The two main functions are   , which can be implemented by ranking branches in the tree of connection options that will be raced , and Protocol Configuration , which happens after the racing is complete and a protocol stack has been chosen . 
 Commit should address all changes requested - please reconsider your reviews 
 As the Application Intents have been moved to the appendix of the API draft for London I would not insert them in the main text for the implementation draft for London . I think it would be better to exemplify the ranking of branches using some of the Protocol and Path Selection Properties . Just one or two examples perhaps ? We could add the application intents to an appendix also for the implementation draft or save it for later .
Paragraph (version A):
The Transport Services architecture I-D.pauly-taps-arch defines a
system that allows applications to use transport networking protocols
flexibly.  This document serves as a guide to implementation on how
to build a system that provides such an interface.
Paragraph (version A):
The Transport Services architecture I-D.pauly-taps-arch defines a
system that allows applications to use transport networking protocols
flexibly.  This document serves as a guide to implementation on how
to build a system that provides such an interface.  The terminology
used in this document is based on the Architecture.
Paragraph (version A):
The interface exposed to applications is defined as the Transport
Services API I-D.trammell-taps-interface.  This API is designed to be
generic across multiple transport protocols and sets of protocols
features.  It is the job of an implementation of a Transport Services
system to turn the requests of an application into decisions on how
to establish connections, and how to transfer data over those
connections once established.
Paragraph (version B):
The Transport Services architecture draft-pauly-taps-arch defines a
system that allows applications to use transport networking protocols
flexibly.  This document serves as a guide to implementation on how
to build a system that provides such an interface.
Paragraph (version B):
The Transport Services architecture draft-pauly-taps-arch defines a
system that allows applications to use transport networking protocols
flexibly.  This document serves as a guide to implementation on how
to build a system that provides such an interface.  The terminology
used in this document is based on the Architecture.
Paragraph (version B):
The interface exposed to applications is defined as the Transport
Services API draft-trammell-taps-interface.  This API is designed to
be generic across multiple transport protocols and sets of protocols
features.  It is the job of an implementation of a Transport Services
system to turn the requests of an application into decisions on how
to establish connections, and how to transfer data over those
connections once established.
Paragraph (version A):
The application requested Transport Parameters which include
requirements or prohibitions that cannot be satisfied by any of
the available protocols.  For example, if an application requires
"Option to configure reliability for individual Content", but no
such protocol is available on the host running the transport
system, e.g., because SCTP is not supported by the operating
system, this should result in an error.
Paragraph (version B):
The application requested Transport Parameters which include
requirements or prohibitions that cannot be satisfied by any of
the available protocols.  For example, if an application requires
"Option to configure reliability for individual Messages", but no
such protocol is available on the host running the transport
system, e.g., because SCTP is not supported by the operating
system, this should result in an error.
Paragraph (version A):
The application requested Transport Parameters which exclude each
other, i.e., the required and prohibited properties cannot be
satisfied by the same protocol.  For example, if an application
prohibits "Reliable Data Transfer" but then requires "Configure
Reliability per Content", this mismatch should result in an error.
Paragraph (version B):
The application requested Transport Parameters which exclude each
other, i.e., the required and prohibited properties cannot be
satisfied by the same protocol.  For example, if an application
prohibits "Reliable Data Transfer" but then requires "Configure
Reliability per Message", this mismatch should result in an error.
Paragraph (version A):
How do the implementation's default policy, a dynamic system policy,
and application preferences interact.
Paragraph (version B):
Application preferences, i.e., preferences specified during the
pre-establishment such as Local Endpoint, Remote Endpoint, Path
Selection Properties, and Protocol Selection Properties.
Paragraph (version B):
Dynamic system policy, i.e., policy compiled from internally and
externally acquired information about available network
interfaces, supported transport protocols, and current/previous
Connections.  Examples of ways to externally retrieve policy-
support information are through OS-specific statistics/measurement
tools and tools that reside on middleboxes and routers.
Paragraph (version B):
Default implementation policy, i.e., predefined policy by OS or
application.
Paragraph (version B):
In general, any protocol or path used for a connection must conform
to all three sources of constraints.  Any violation of any of the
layers should cause a protocol or path to be considered ineligble for
use.  For an example of application preferences leading to
constraints, an application may prohibit the use of metered network
interfaces for a given Connection to avoid user cost.  Similarly, the
system policy at a given time may prohibit the use of such a metered
network interface from the application's process.  Lastly, the
implementation itself may default to disallowing certain network
interfaces unless explicitly requested by the application and allowed
by the system.
Paragraph (version B):
It is expected that the database of system policies and the method of
looking up these policies will vary across various platforms.  An
implementation SHOULD attempt to look up the relevant policies for
the system in a dynamic way to make sure it is reflecting an accurate
version of the system policy, since the system's policy regarding the
application's traffic may change over time due to user or
administrative changes.
Paragraph (version B):
Implementations SHOULD sort the branches of the tree of connection
options in order of their preference rank.  Leaf nodes on branches
with higher rankings represent connection attempts that will be raced
first.  Implementations SHOULD order the branches to reflect the
preferences expressed by the application for its new connection,
including Protocol and Path Selection Properties, which are specified
in draft-trammell-taps-interface In addition to the properties
provided by the application, an implementation MAY include additional
criteria such as cached performance estimates, see performance-
caches, or system policy, see role-of-system-policy, in the ranking.
Paragraph (version B):
Interface Type: If the application specifies an interface type to
be preferred or avoided, implementations SHOULD rank paths
accordingly.  If the application specifies an interface type to be
required or prohibited, we expect an implementation to not include
the non-conforming paths into the three.
Paragraph (version A):
Implementations may select the criteria by which a leaf node is
considered to be successfully connected differently on a per-protocol
basis.  If the only protocol being used is a transport protocol with
a clear handshake, like TCP, then the obvious choice is to declare
that node "connected" when the last packet of the three-way handshake
has been received.  If the only protocol being used is an
"unconnected" protocol, like UDP, the implementation may consider the
node fully "connected" the moment it determines a route is present,
before sending any packets on the network.
Paragraph (version B):
Implementations may select the criteria by which a leaf node is
considered to be successfully connected differently on a per-protocol
basis.  If the only protocol being used is a transport protocol with
a clear handshake, like TCP, then the obvious choice is to declare
that node "connected" when the last packet of the three-way handshake
has been received.  If the only protocol being used is an
"unconnected" protocol, like UDP, the implementation may consider the
node fully "connected" the moment it determines a route is present,
before sending any packets on the network unconnected-racing.
Paragraph (version A):
When the Connections that are offered to an application by the
Transport System are multiplexed, the Transport System may implement
the establishment of a new Connection by simply beginning to use a
new stream of an already established transport connection.  This,
then, means that there may not be any "establishment" message (like a
TCP SYN), but the application can simply start sending or receiving.
Therefore, when a Transport System's "Initiate" Action is called
without Content being handed over, it cannot be guaranteed that the
other endpoint will have any way to know about this, and hence a
passive endpoint's ConnectionReceived event may not be called upon an
active endpoint's Inititate.  Instead, calling the ConnectionReceived
event may be delayed until the first Content arrives.
Paragraph (version A):
How to handle UDP, and allowing the application or anothe protocol to
cause the next option to be tried.
Paragraph (version A):
How to handle sending data onto examples like TCP, UDP, and a basic
Length-Value protocol.
Paragraph (version A):
How to handle and notify errors when sending.
Paragraph (version A):
How to handle sending data in examples like TCP, UDP, and a basic
Length-Value protocol.
Paragraph (version A):
Waiting for frame boundaries when necessary.
Paragraph (version A):
How to handle and notify errors when receiving.
Paragraph (version B):
When the Connections that are offered to an application by the
Transport System are multiplexed, the Transport System may implement
the establishment of a new Connection by simply beginning to use a
new stream of an already established transport connection.  This,
then, means that there may not be any "establishment" message (like a
TCP SYN), but the application can simply start sending or receiving.
Therefore, when a Transport System's "Initiate" Action is called
without Messages being handed over, it cannot be guaranteed that the
other endpoint will have any way to know about this, and hence a
passive endpoint's ConnectionReceived event may not be called upon an
active endpoint's Inititate.  Instead, calling the ConnectionReceived
event may be delayed until the first Message arrives.
Paragraph (version B):
While protocols that use an explicit handshake to validate a
Connection to a peer can be used for racing multiple establishment
attempts in parallel, "unconnected" protocols such as raw UDP do not
offer a way to validate the presence of a peer or the usability of a
Connection without application feedback.  An implementation SHOULD
consider such a protocol stack to be established as soon as a local
route to the peer endpoint is confirmed.
Paragraph (version B):
However, if a peer is not reachable over the network using the
unconnected protocol, or data cannot be exchanged for any other
reason, the application may want to attempt using another candidate
Protocol Stack.  The implementation SHOULD maintain the list of other
candidate Protocol Stacks that were eligible to use.  In the case
that the application signals that the initial Protocol Stack is
failing for some reason and that another option should be attempted,
the Connection can be updated to point to the next candidate Protocol
Stack.  This can be viewed as an application-driven form of Protocol
Stack racing.
Paragraph (version B):
Unconnected protocols such as UDP and UDP-lite generally do not
provide the same mechanisms that connected protocols do to offer
Connection objects.
Paragraph (version B):
Implementations should wait for incoming packets for unconnected
protocols on a listening port and should perform five-tuple matching
of packets to either existing Connection objects or the creation of
new Connection objects.  On platforms with facilities to create a
"virtual connection" for unconnected protocols implementations should
use these mechanisms to minimise the handling of datagrams intended
for already created Connection objects.
Paragraph (version B):
The most basic mapping for sending a Message is an abstraction of
datagrams, in which the transport protocol naturally deals in
discrete packets.  Each Message here corresponds to a single
datagram.  Generally, these will be short enough that sending and
receiving will always use a complete Message.
Paragraph (version B):
For protocols that expose byte-streams, the only delineation provided
by the protocol is the end of the stream in a given direction.  Each
Message in this case corresponds to the entire stream of bytes in a
direction.  These Messages may be quite long, in which case they can
be sent in multiple parts.
Paragraph (version B):
Protocols that provide the framing (such as length-value protocols,
or protocols that use delimeters) provide data boundaries that may be
longer than a traditional packet datagram.  Each Message for framing
protocols corresponds to a single frame, which may be sent either as
a complete Message, or in multiple parts.
Paragraph (version B):
The effect of the application sending a Message is determined by the
top-level protocol in the established Protocol Stack.  That is, if
the top-level protocol provides an abstraction of framed messages
over a connection, the application will be able to send multiple
Messages on that connection, even if the framing protocol is built on
a byte-stream protocol like TCP.
Paragraph (version B):
Lifetime should be implemented by removing the Message from its
queue of pending Messages after the Lifetime has expired.  A queue
of pending Messages within the transport system implementation
that have yet to be handed to the Protocol Stack can always
support this property, but once a Message has been sent into the
send buffer of a protocol, only certain protocols may support de-
queueing a message.  For example, TCP cannot remove bytes from its
send buffer, while in case of SCTP, such control over the SCTP
send buffer can be exercised using the partial reliability
extension RFC8303.  When there is no standing queue of Messages
within the system, and the Protocol Stack does not support
removing a Message from its buffer, this property may be ignored.
Paragraph (version B):
Niceness represents the ability to de-prioritize a Message in
favor of other Messages.  This can be implemented by the system
re-ordering Messages that have yet to be handed to the Protocol
Stack, or by giving relative priority hints to protocols that
support priorities per Message.  For example, an implementation of
HTTP/2 could choose to send Messages of different niceness on
streams of different priority.
Paragraph (version B):
Ordered: when this is false, it disables the requirement of in-
order-delivery for protocols that support configurable ordering.
Paragraph (version B):
Idempotent: when this is true, it means that the Message can be
used by mechanisms that might transfer it multiple times - e.g.,
as a result of racing multiple transports or as part of TCP Fast
Open.
Paragraph (version B):
Corruption Protection Length: when this is set to any value other
than -1, it limits the required checksum in protocols that allow
limiting the checksum length (e.g.  UDP-Lite).
Paragraph (version B):
Immediate Acknowledgement: this informs the implementation that
the sender intends to execute tight control over the send buffer,
and therefore wants to avoid delayed acknowledgements.  In case of
SCTP, a request to immediately send acknowledgements can be
implemented using the "sack-immediately flag" described in
Section 4.2 of RFC8303 for the SEND.SCTP primitive.
Paragraph (version B):
Instantaneous Capacity Profile: when this is set to "Interactive/
Low Latency", the Message should be sent immediately, even when
this comes at the cost of using the network capacity less
efficiently.  For example, small messages can sometimes be bundled
to fit into a single data packet for the sake of reducing header
overhead; such bundling SHOULD not be used in case this parameter
is true.  For example, in case of TCP, the Nagle algorithm SHOULD
be disabled when Immediate=true.  Scavenger/Bulk can translate
into usage of a congestion control mechanism such as LEDBAT, and/
or the capacity profile can lead to a choice of a DSCP value as
described in I-D.ietf-taps-minset).
Paragraph (version B):
The application should be notified whenever a Message or partial
Message has been consumed by the Protocol Stack, or has failed to
send.  This meaning of the Message being consumed by the stack may
vary depending on the protocol.  For a basic datagram protocol like
UDP, this may correspond to the time when the packet is sent into the
interface driver.  For a protocol that buffers data in queues, like
TCP, this may correspond to when the data has entered the send
buffer.
Paragraph (version B):
Similar to sending, Receiving a Message is determined by the top-
level protocol in the established Protocol Stack.  The main
difference with Receiving is that the size and boundaries of the
Message are not known beforehand.  The application can communicate in
its Receive action the parameters for the Message, which can help the
implementation know how much data to deliver and when.  For example,
if the application only wants to receive a complete Message, the
implementation should wait until an entire Message (datagram, stream,
or frame) is read before delivering any Message content to the
application.  Alternatively, the application can specify the minimum
number of bytes of Message content it wants to receive (which may be
just a single byte) to control the flow of received data.
Paragraph (version B):
If a Connection becomes finished before a requested Receive action
can be satisfied, the implementation should deliver any partial
Message content outstanding, or if none is available, an indication
that there will be no more received Messages.
Paragraph (version B):
Timeout for aborting Connection: for SCTP, this can be done using
the primitive CHANGE_TIMEOUT.SCTP described in section 4 of
RFC8303.
Paragraph (version B):
Abort timeout to suggest to the Remote Endpoint: for TCP, this can
be done using the primitive CHANGE_TIMEOUT.TCP described in
section 4 of RFC8303.
Paragraph (version B):
Retransmission threshold before excessive retransmission
notification: for TCP, this can be done using ERROR.TCP described
in section 4 of RFC8303.
Paragraph (version B):
Required minimum coverage of the checksum for receiving: for UDP-
Lite, this can be done using the primitive
SET_MIN_CHECKSUM_COVERAGE.UDP-Lite described in section 4 of
RFC8303.
Paragraph (version B):
Connection group transmission scheduler: for SCTP, this can be
done using the primitive SET_STREAM_SCHEDULER.SCTP described in
section 4 of RFC8303.
Paragraph (version A):
Beyond a single connection's lifetime, it is useful to keep state and
history.
Paragraph (version A):
Caching for DNS, TLS, etc.  Associated with sets of endpoints for
future use.
Paragraph (version A):
Caching of round trip time (RTT), success rate with various protocols
and features.
Paragraph (version B):
Beyond a single Connection's lifetime, it is useful for an
implementation to keep state and history.  This cached state can help
improve future Connection establishment due to re-using results and
credentials, and favoring paths and protocols that performed well in
the past.
Paragraph (version B):
Cached state may be associated with different Endpoints for the same
Connection, depending on the protocol generating the cached content.
For example, session tickets for TLS are associated with specific
endpoints, and thus SHOULD be cached based on a Connection's hostname
Endpoint (if applicable).  On the other hand, performance
characteristics of a path are more likely tied to the IP address and
subnet being used.
Paragraph (version B):
Some protocols will have long-term state to be cached in association
with Endpoints.  This state often has some time after which it is
expired, so the implementation SHOULD allow each protocol to specify
an expiration for cached content.
Paragraph (version B):
The DNS protocol can cache resolution answers (A and AAAA queries,
for example), associated with a Time To Live (TTL) to be used for
future hostname resolutions without requiring asking the DNS
resolver again.
Paragraph (version B):
TLS caches session state and tickets based on a hostname, which
can be used for resuming sessions with a server.
Paragraph (version B):
TCP can cache cookies for use in TCP Fast Open.
Paragraph (version B):
Cached protocol state is primarily used during Connection
establishment for a single Protocol Stack, but may be used to
influence an implementation's preference between several candidate
Protocol Stacks.  For example, if two IP address Endpoints are
otherwise equally preferred, an implementation may choose to attempt
a connection to an address for which it has a TCP Fast Open cookie.
Paragraph (version B):
Applications must have a way to flush protocol cache state if
desired.  This may be necessary, for example, if application-layer
identifiers rotate and clients wish to avoid linkability via
trackable TLS tickets or TFO cookies.
Paragraph (version B):
These items can be cached on a per-address and per-subnet
granularity, and averaged between different values.  The information
SHOULD be cached on a per-network basis, since it is expected that
different network attachments will have different performance
characteristics.  Besides Protocol Instances, other system entities
may also provide data into performance-oriented caches.  This could
for instance be signal strength information reported by radio modems
like Wi-Fi and mobile broadband or information about the battery-
level of the device.  Furthermore, the system may cache the observed
maximum throughput on a path as an estimate of the available
bandwidth.
Paragraph (version B):
An implementation should use this information, when possible, to
determine preference between candidate paths, endpoints, and protocol
options.  Eligible options that historically had significantly better
performance than others SHOULD be selected first when gathering
candidates gathering to ensure better performance for the
application.
Paragraph (version B):
It is recognized that consistency in caching becomes a major issue
for performance caches, since if a cached value is not valid the
whole optimization might be lost.  Still, recommendations on cache
lifetimes are difficult to offer since it largely depends on the
variability of the cached performance information.  For example, the
round-trip times experienced by TCP segments over a particular
network path might vary a lot over quite short time intervals.  Thus,
for this kind of performance information the cache lifetime should be
kept short, e.g., a fixed number of RTTs.  In contrast, performance
information such as the connection establishment success rate, could
probably have a cache lifetime that spans hours.
Paragraph (version B):
Connection lifetime for TCP translates fairly simply into the the
abstraction presented to an application.  When the TCP three-way
handshake is complete, its layer of the Protocol Stack can be
considered Ready (established).  This event will cause racing of
Protocol Stack options to complete if TCP is the top-level protocol,
at which point the application can be notified that the Connection is
Ready to send and receive.
Paragraph (version B):
If the application sends a Close, that can translate to a graceful
termination of the TCP connection, which is performed by sending a
FIN to the remote endpoint.  If the application sends an Abort, then
the TCP state can be closed abruptly, leading to a RST being sent to
the peer.
Paragraph (version B):
Without a layer of framing above TCP, the cleanest abstraction for
sending and receiving Messages over TCP is to treat each direction of
the stream of bytes as a single Message, terminated by a FIN.  That
means that if the application can make several Send calls to enqueue
subsequent data chunks for the same Message to continue writing, but
marking the Message complete corresponds to closing the sending
stream.  Similarly, when the application receives the final portion
of a Message, it knows that the receiving stream has been closed.
Paragraph (version B):
UDP as a direct transport does not provide any handshake or
connectivity state, so the notion of the transport protocol becoming
Ready or established is degenerate.  Once the system has validated
that there is a route on which to send and receive UDP datagrams, the
protocol is considered Ready.  Similarly, a Close or Abort has no
meaning to the on-the-wire protocol, but simply leads to the local
state being torn down.
Paragraph (version B):
When sending and receiving messages over UDP, each Message should
correspond to a single UDP datagram.  The Message can contain
metadata about the packet, such as the ECN bits applied to the
packet.
Paragraph (version A):
When a Transport System transfers Content, it seems natural to map
Content to SCTP messages in order to support properties such as
"Ordered" or "Lifetime" (which maps onto partially reliable delivery
with a SCTP_PR_SCTP_TTL policy RFC6458).  However, since multiplexing
of Connections onto SCTP streams may happen, and would be hidden from
the application, the Transport System requires a per-stream receiver
buffer anyway, so this potential benefit is lost and the "partial
flag" becomes unnecessary for the system.
Paragraph (version B):
When a Transport System transfers a Message, it seems natural to map
the Message object to SCTP messages in order to support properties
such as "Ordered" or "Lifetime" (which maps onto partially reliable
delivery with a SCTP_PR_SCTP_TTL policy RFC6458).  However, since
multiplexing of Connections onto SCTP streams may happen, and would
be hidden from the application, the Transport System requires a per-
stream receiver buffer anyway, so this potential benefit is lost and
the "partial flag" becomes unnecessary for the system.
Paragraph (version B):
The mapping of a TLS stream abstraction into the application is
equivalent to the contract provided by TCP tcp.  The Ready state
should be determined by the completion of the TLS handshake, which
involves potentially several more round trips beyond the TCP
handshake.  The application should not be notified that the
Connection is Ready until TLS is established.
Paragraph (version B):
HTTP requests and responses map naturally into Messages, since they
are delineated chunks of data with metadata that can be sent over a
transport.  To that end, HTTP can be seen as the most prevalent
framing protocol that runs on top of streams like TCP, TLS, etc.
Paragraph (version B):
In order to use a transport Connection that provides HTTP Message
support, the establishment and closing of the connection can be
treated as it would without the framing protocol.  Sending and
receiving of Messages, however, changes to treat each Message as a
well-delineated HTTP request or response, with the content of the
Message representing the body, and the Headers being provided in
Message metadata.
Paragraph (version B):
QUIC provides a multi-streaming interface to an encrypted transport.
Each stream can be viewed as equivalent to a TLS stream over TCP, so
a natural mapping is to present each QUIC stream as an individual
Connection.  The protocol for the stream will be considered Ready
whenever the underlying QUIC connection is established to the point
that this stream's data can be sent.  For streams after the first
stream, this will likely be an immediate operation.
Paragraph (version B):
Closing a single QUIC stream, presented to application as a
Connection, does not imply closing the underlying QUIC connection
itself.  Rather, the implementation may choose to close the QUIC
connection once all streams have been closed (possibly after some
timeout), or after an individual stream Connection sends an Abort.
Paragraph (version B):
Messages over a direct QUIC stream should be represented similarly to
the TCP stream tcp (one Message per direction), unless a framing
mapping is used on top of QUIC.
Paragraph (version B):
Similar to QUIC quic, HTTP/2 provides a multi-streaming interface.
This will generally use HTTP as the unit of Messages over the
streams, in which each stream can be represented as a transport
Connection.  The lifetime of streams and the HTTP/2 connection should
be managed as described for QUIC.
Paragraph (version B):
It is possible to treat each HTTP/2 stream as a raw byte-stream
instead of carrier for HTTP messages, in which case the Messages over
the streams can be represented similarly to the TCP stream tcp (one
Message per direction).