Draft: draft-ietf-detnet-data-plane-framework

Commentary: Branch for review 
 Merge it ! 
 Changes look good - clicking correct button this time
Paragraph (version A):
Deterministic Networking (DetNet) is a service that can be offered by
a network to DetNet flows.  DetNet provides these flows with low
packet loss rates and assured maximum end-to-end delivery latency.  A
description of the general background and concepts of DetNet can be
found in I-D.ietf-detnet-architecture.
Paragraph (version B):
Deterministic Networking (DetNet) is a service that can be offered by
a network to DetNet flows.  DetNet provides these flows with
extremely low packet loss rates and assured maximum end-to-end
delivery latency.  A description of the general background and
concepts of DetNet can be found in I-D.ietf-detnet-architecture.
Paragraph (version A):
The DetNet Architecture decomposes the DetNet related data plane
functions into two sub-layers: a service sub-layer and a forwarding
sub-layer.  The service sub-layer is used to provide DetNet service
protection and reordering.  The forwarding sub-layer is used to
provide congestion protection (low loss, assured latency, and limited
reordering) and leverages Traffic Engineering mechanisms.
Paragraph (version B):
The DetNet Architecture models the DetNet related data plane
functions decomposed into two sub-layers: a service sub-layer and a
forwarding sub-layer.  The service sub-layer is used to provide
DetNet service protection and reordering.  The forwarding sub-layer
is used to provide congestion protection (low loss, assured latency,
and limited reordering) and leverages Traffic Engineering mechanisms.
Paragraph (version A):
DetNet flows may be carried over network technologies that can
provide the DetNet required level of service.  For example, DetNet
MPLS flows can be carried over IEEE 802.1 Time Sensitive Network
(TSN) sub-networks.  However, IEEE 802.1 TSN support is not required
and some of the DetNet benefits can be gained by running over a data
link layer that has not been specificaly enhanced to support TSN.
Paragraph (version A):
Different traffic types, or application flows, can be mapped on top
of DetNet.  DetNet can optionally reuse header information provided
by, or shared with, applications.  An example of such can be found in
I-D.ietf-detnet-ip.
Paragraph (version B):
DetNet flows may be carried over network technologies that can
provide the DetNet required level of service.  For example, DetNet
MPLS flows can be carried over IEEE 802.1 Time Sensitive Network
(TSN) IEEE802.1TSNTG sub-networks.  However, IEEE 802.1 TSN support
is not required and some of the DetNet benefits can be gained by
running over a data link layer that has not been specifically
enhanced to support TSN.
Paragraph (version B):
Different traffic types, or application flows, can be mapped on top
of DetNet.  DetNet can optionally reuse header information provided
by, or shared with, applications.  An example of shared header fields
can be found in I-D.ietf-detnet-ip.
Paragraph (version A):
The DetNet forwarding layer may be directly provided by the DetNet
forwarding layer, for example by IP or MPLS.  Alternatively an
overlay approach may be used in which the packet is natively carried
between key nodes within the DetNet network (say between PREOF nodes)
and a sublayer is used to provide the information needed to reach the
next hop in the overlay.
Paragraph (version A):
This forwarding layer provides the quality underpin needed by the
DetNet Service layer.  It may do this directly through the use of
queuing techniques and traffic engineering methods, or it may do this
through the assistance of its underlying connectivity.  For example
it may call upon IEEE 802.1 TSN [NEEDS REFERENCE].
Paragraph (version A):
The service layer provides additional support beyond the connectivity
function of the forwarding layer.  An example of this is Packet
Replication, Elimination, and Ordering (PREOF) function [ADD REF].
Paragraph (version A):
Whether the DetNet flows are carried in there native form across
the DetNet network, or whether they are encapsulated.
Paragraph (version A):
Whether or not the DetNet data plane incudes the facility to carry
additional information (metadata) that can be used to provide an
enhanced service to the DetNet packet.
Paragraph (version A):
The encapsulation of the DetNet flows allows them to be sent over a
data plane of a type other than their native type.  Encapsulation is
essential if, for example, it is required to sent Ethernet DetNet
flows over a data plane such as MPLS.
Paragraph (version A):
The use of encapsulation is also required if additional information
(meta-data) is needed by the DetNet data plane and their is either no
ability to include it in the client data packet, or the specification
of the client data plane do not permit the modification of the packet
to include additional data.  An example of such meta-data is the
inclusion of a sequence number required by the PREOF function.
Paragraph (version A):
Encalsulation is also needed if the DetNet flow or agregate flow is
not easily recognised from from its native encapsulation.
Paragraph (version A):
Metadata can be a useful way of identifying packets that need to be
treated as a flow or flow agregate.  It is also useful as a way of
including a sequence number the packet for use by the PREOF function
or as a place to carry OAM indications or OAM information to
instrument DetNet data plane operation.
Paragraph (version A):
An IP data plane may operate natively or through the use of an
encapsulation.  There are many IP encapsulations that may be
interposed between the DetNet data plane IP header and the DetNet
payload, and it is anticipated that more than one encapulation may be
deployed.
Paragraph (version B):
ProtStack1 reproduced from the I-D.ietf-detnet-architecture,shows a
logical DetNet service with the two sub-layers.
Paragraph (version B):
The DetNet forwarding sub-layer may be directly provided by the
DetNet service layer, for example by IP, MPLS or IEEE 802.1 TSN
IEEE802.1TSNTG.  Alternatively an overlay approach may be used in
which the packet is natively carried between key nodes within the
DetNet network (say between PREOF nodes) and a sublayer is used to
provide the information needed to reach the next hop in the overlay.
Paragraph (version B):
This forwarding layer provides the quality underpin needed by the
DetNet Service layer.  It may do this directly through the use of
queuing techniques and traffic engineering methods, or it may do this
through the assistance of its underlying connectivity.  For example
it may call upon Ethernet TSN capabilities defined in IEEE 802.1 TSN
IEEE802.1TSNTG.
Paragraph (version B):
The service layer provides additional support beyond the connectivity
function of the forwarding layer.  An example of this is Packet
Replication, Elimination, and Ordering (PREOF) function see PREOF.
Paragraph (version B):
Whether the DetNet flows are carried in there native form across
the DetNet network, or whether they are encapsulated.  dn_svcs and
dn_svc_encaps contrast native services and the encapsulation of
services.
Paragraph (version B):
Whether or not the DetNet data plane includes the facility to
carry additional information (metadata) that can be used to
provide an enhanced service to the DetNet packet.
Paragraph (version B):
The encapsulation of the DetNet flows allows them to be sent over a
data plane of a type other than their native type.  Encapsulation is
essential if, for example, it is required to send Ethernet DetNet
flows over a data plane such as MPLS.  dn_svc_encaps illustrates some
encapsulation combinations.
Paragraph (version B):
The use of encapsulation is also required if additional information
(meta-data) is needed by the DetNet data plane and their is either no
ability to include it in the client data packet, or the specification
of the client data plane does not permit the modification of the
packet to include additional data.  An example of such meta-data is
the inclusion of a sequence number required by the PREOF function.
Paragraph (version B):
Encapsulation is also needed if the DetNet flow or aggregate flow is
not easily recognised from from its native encapsulation.
Paragraph (version B):
Metadata can be a useful way of identifying packets that need to be
treated as a flow or flow aggregate.  It is also useful as a way of
including a sequence number the packet for use by the PREOF function
or as a place to carry OAM indications or OAM information to
instrument DetNet data plane operation.
Paragraph (version B):
Explicit inclusion of metadata is possible through the use of IP
options or IP extension headers.  New IP options are almost
impossible to get standardized or to deploy in an operational network
and will not be discussed further in this text.  IPv6 extensions
headers are finding popularity in current IPv6 development work,
particularly in connection with Segment Routing of IPv6 (SRv6) and IP
OAM.  The design of a new IPv6 extension header or the modification
of an existing one is a technique available in the tool box of the
DetNet IP data plane designer.
Paragraph (version B):
Explicit inclusion of metadata in an IP packet is also possible
through the inclusion of an MPLS label stack and the MPLS DetNet
Control Word using one of the methods for carrying MPLS over IP I-
D.mpls-over-udp-ip.  This is described in more detail in
subnet_considerations.
Paragraph (version B):
Implicit metadata can be included through the use of the network
programming paradigm I-D.spring-srv6-network-programming in which the
suffix of an IPv6 address is used to encode additional information
for use by the network of the receiving host.  Examples of such
information include the sequence number for use by the PREOF
function, or even all the essential information being deigned into
the DetNet over MPLS label stack (the DetNet Control Word and the
DetNet Service label).
Paragraph (version B):
An IP data plane may operate natively or through the use of an
encapsulation.  There are many IP encapsulations that may be
interposed between the DetNet data plane IP header and the DetNet
payload, and it is anticipated that more than one encapsulation may
be deployed.
Paragraph (version A):
One method of operating an IP DetNet data plane without encapsulation
is to use "6-tuple" based flow identification, where "6-tuple" refers
to information carried in IP and higher layer protocol headers.
General background on the use of IP headers, and "5-tuples", to
identify flows and support Quality of Service (QoS) can be found in
RFC3670.  RFC7657 also provides useful background on the delivery
differentiated services (DiffServ) and "6-tuple" based flow
identification.  DetNet flow aggregation may be enabled via the use
of wildcards, masks, prefixes and ranges.  The operation of this
method is described in detail in [ID.ietf-detnet-ip].
Paragraph (version A):
The DetNet forwarding plane may use explict route cabability and
traffic engineering capabilities to provide a forwarding sub-layer
that is responsible for providing resource allocation and explicit
routes.  It is possible to include metadata in a native IP packet
explicitly, or implicitly.
Paragraph (version A):
Explicit inclusion of metadata is possible through the use of IP
options or IP extension headers.  New IP options are almost
impossible to get standardized or to deploy in an operational network
and will not be discussed further in this text.  IPv6 extensions
headers are finding popularity in current IPv6 development work,
particularly in connection with Segment Routing of IPv6 (SRv6) and
IOAM.  The design of a new IPv6 extension header or the modification
of an existing one is a technique in the tool box of the DetNet IP
data plane designer.
Paragraph (version A):
Explicit inclusion of metadata in an IP packet is also possible
through the inclusion of an MPLS label stack and the MPLS DetNet
Control Word using one of the methods for carrying MPLS over IP [NEED
REFERENCES].  This is described in more detail in [NEED REFERENCE].
Paragraph (version A):
Implicit metadata can be included through the use of the network
programming paradygm [NEED REFERENCE] in which the suffix of an IPv6
address is used to encode additional information for use by the
network of the receiving host.  Examples of such information include
the sequence number for use by the PREOF function, or indeed the
whole of the essential information being deigned into the DetNet over
MPLS label stack (the DetNet Control Word and the DetNet Service
lable).
Paragraph (version A):
dn_fmwrk_stacks illustrates the service stack combinations for
DetNet.  At the service layer IP, MPLS and TSN (as an L2 VPN) offer
detnet services.  (TSN is also a native L2 DetNet Service covered in
IEEE8021Q.)  These services can be stacked as illustrated.  For
example, IP DetNet can be delivered purely as an IP service.  Or IP
DetNet can leverage MPLS for service and forwarding or just MPLS
forwaring.  As a result IP DenNet can have acces to MPLS traffic
Engineering.  Alternatively IP DetNet can leverage TSN by leveraging
Ethenret with TSN.  The diagram shows several common permutations.
Paragraph (version A):
MPLS provides the abbility to forward traffic over implicit and
explicit paths to the point in the network where the next DetNet
service layer action needs to take place.  It does this through the
use of a stack of one or more labels with various forwarding
semantics.
Paragraph (version A):
MPLS also provides the ability to identify a service instance that is
to process the packet through the use of a label that maps the packet
to a service instance.
Paragraph (version A):
In cases where metadata is needed to process an MPLS encapsulated
packet at the service layer, this has been been provided through the
use of a shim layer also called a control word (CW) RFC4385.
Although such CWs are requently 32 bits long, there is no
architectural constraint on its size of this structure, only the
requirement that it is fully understood by all parties operating on
it in the DetNet service layer.
Paragraph (version B):
One method of operating an IP DetNet data plane without encapsulation
is to use "6-tuple" based flow identification, where "6-tuple" refers
to information carried in IP and higher layer protocol headers.
General background on the use of IP headers, and "5-tuples", to
identify flows and support Quality of Service (QoS) can be found in
RFC3670.  RFC7657 also provides useful background on the delivery
differentiated services (DiffServ) and "tuple" based flow
identification.  DetNet flow aggregation may be enabled via the use
of wildcards, masks, prefixes and ranges.  The operation of this
method is described in detail in I-D.ietf-detnet-ip.
Paragraph (version B):
The DetNet forwarding plane may use explicit route capabilities and
traffic engineering capabilities to provide a forwarding sub-layer
that is responsible for providing resource allocation and explicit
routes.  It is possible to include metadata in a native IP packet
explicitly, or implicitly.
Paragraph (version B):
MPLS provides the ability to forward traffic over implicit and
explicit paths to the point in the network where the next DetNet
service layer action needs to take place.  It does this through the
use of a stack of one or more labels with various forwarding
semantics.
Paragraph (version B):
MPLS also provides the ability to identify a service instance that is
used to process the packet through the use of a label that maps the
packet to a service instance.
Paragraph (version B):
In cases where metadata is needed to process an MPLS encapsulated
packet at the service layer, this has been been provided through the
use of a shim layer also called a control word (CW) RFC4385.
Although such CWs are frequently 32 bits long, there is no
architectural constraint on its size of this structure, only the
requirement that it is fully understood by all parties operating on
it in the DetNet service layer.  The operation of this method is
described in detail in I-D.ietf-detnet-mpls.
Paragraph (version A):
Use of allocated resources (queuing, policing, shaping) to ensure
that the congestion-related loss and latency/jitter requirements
of a DetNet flow are met.
Paragraph (version A):
Use of a specific path for a flow.  This limits misordering and
bounds latency.
Paragraph (version A):
Which in the case of this document primarily relates to
replication and elimination.  Changing the explicit path after a
failure is detected in order to restore delivery of the required
DetNet service characteristics is also possible.  Path changes,
even in the case of failure recovery, can lead to the out of order
delivery of data.
Paragraph (version A):
Generally, distributing packets of the same DetNet flow over
multiple paths is not recommended.  Such load sharing, e.g., via
ECMP or UCMP, impacts ordering and possibly jitter.
Paragraph (version B):
Reservation of resources can allocate resources to specific DetNet
flows.  This can eliminate packet contention and loss for DetNet
traffic.  This also can reduce jitter for the DetNet traffic.
DetNet flows are assumed to behave with respect to the reserved
traffic profile.  If other traffic shares the link resources, the
use of (queuing, policing, shaping) policies can be used to ensure
that the allocation of resources reserved for DetNet is met.
Queuing and shaping of DetNet traffic could be required to ensure
that DetNet traffic does not exceed its reserved profile but this
would impact the DetNet service characteristics.
Paragraph (version B):
Use of a specific path for a flow.  This allows control of the
network delay by steering the packet with the ability to influence
the physical path.  Explicit routes complement reservation by
ensuring that a consistent path can be associated with its
resources for the duration of that path.  Coupled with the traffic
mechanism, this limits misordering and bounds latency.
Paragraph (version B):
Use of multiple packet streams using multiple paths, for example
1+1 or 1:1 linear protection.  For DetNet this primarily relates
to packet replication and elimination capabilities.  Changing the
explicit path after a failure is detected to an already
established path in order to restore delivery of the required
DetNet service characteristics is another protection mechanism for
example MPLS hitless protection.  Path changes, even in the case
of failure recovery, can lead to the out of order delivery of data
requiring packet ordering functions either within the DetNet
service or at a high layer in the application traffic.
Establishment of new paths after a failure is out of scope for
DetNet services.
Paragraph (version B):
Use of network coding to improve a network's throughput,
efficiency, latency, and scalability, as well as resilience to
partition, attacks, and eavesdropping, as compared to traditional
methods.  Network coding is another technique that uses multiple
paths typically with very similar delays and capacity.  Network
coding is often applied in wireless networks.
Paragraph (version B):
Use of packet by packet distribution of the same DetNet flow over
multiple paths is not recommended.  Such load sharing, e.g., via
ECMP or UCMP, impacts ordering and possibly jitter.
Paragraph (version A):
The DetNet data plane also allows for the aggregation of DetNet
flows, to improved scaling.  How this is done is data plane
dependent.  When DetNet flows are aggregated, transit nodes provide
service to the aggregate and not on a per-DetNet flow basis.  In this
case, nodes performing aggregation will ensure that per-flow service
requirements are achieved.
Paragraph (version B):
Several of these capabilities are expanded upon in more detail below.
Paragraph (version B):
Service protection allow DetNet services to increase reliability and
maintain a DetNet Service Assurance in the case of network congestion
or some failures.  Detnet relies on the underlying technology
capabilities for various protection schemes.  Protection schemes
enable partial or complete coverage of the network paths and active
protection with combinations of PRF, PRE, and POF.
Paragraph (version B):
An example DetNet MPLS network fragment and packet flow is
illustrated in dn_protection_flow.
Paragraph (version B):
In dn_protection_flow the numbers are used to identify the instance
of a packet.  Packet 1 is the original packet, and packets 1.1, and
1.2 are two first generation copies of packet 1.  Packet 1.2.1 is a
second generation copy of packet 1.2 etc.  Note that these numbers
never appear in the packet, and are not to be confused with sequence
numbers, labels or any other identifier that appears in the packet.
They simply indicate the generation number of the original packet so
that its passage through the network fragment can be identified to
the reader.
Paragraph (version B):
Customer Equipment CE1 sends a packet into the DetNet enabled
network.  This is packet (1).  Edge Node EN1 encapsulates the packet
as a DetNet Packet and sends it to Relay node R1 (packet 1.1).  EN1
makes a copy of the packet (1.2), encapsulates it and sends this copy
to Relay node R4.
Paragraph (version B):
Note that along the path from EN1 to R1 there may be zero or more
nodes which, for clarity, are not shown.  The same is true for any
other path between two DetNet entities shown in dn_protection_flow .
Paragraph (version B):
Relay node R4 has been configured to send one copy of the packet to
Relay Node R2 (packet 1.2.1) and one copy to Edge Node EN2 (packet
1.2.2).
Paragraph (version B):
R2 receives packet copy 1.2.1 before packet copy 1.1 arrives, and,
having been configured to perform packet elimination on this DetNet
flow, forwards packet 1.2.1 to Relay Node R3.  Packet copy 1.1 is of
no further use and so is discarded by R2.
Paragraph (version B):
Edge Node EN2 receives packet copy 1.2.2 from R4 before it receives
packet copy 1.2.1 from R2 via relay Node R3.  EN2 therefore strips
any DetNet encapsulation from packet copy 1.2.2 and forwards the
packet to CE2.  When EN2 receives the later packet copy 1.2.1 this is
discarded.
Paragraph (version B):
The above is of course illustrative of many network scenarios that
can be configured.  Between a pair of relay nodes there may be one or
more transit nodes that simply forward the DetNet traffic, but these
are omitted for clarity.
Paragraph (version B):
This example also illustrates 1:1 protection scheme meaning there is
traffic and path for each segment of the end to end path.  Local
DetNet relay nodes determine which packets are eliminated and which
packets are forwarded.  A 1+1 scheme where only one path is used for
traffic at a time, could use the same topology.  In this case there
is no PRF function and traffic is switched upon detection of failure.
An OAM scheme that monitors the paths detects the loss of path or
traffic is required to initiate the switch.  A POF may still be used
in this case to prevent misordering of packets.  In both cases the
protection paths are established and maintained for the duration of
the DetNet service.
Paragraph (version B):
Ring protection may also be supported if the underlying technology
supports it.  Many of the same concepts apply however Rings are
normally 1+1 protection for data efficiency reasons.  RFC8227 is an
example of MPLS-TP data plane that supports Ring protection.
Paragraph (version B):
The DetNet data plane also allows for the aggregation of DetNet
flows, to improved scaling by reducing the state per hop.  How this
is done is data plane or control plane dependent.  When DetNet flows
are aggregated, transit nodes provide service to the aggregate and
not on a per-DetNet flow basis.  When aggregating DetNet flows the
flows should be compatible i.e. the same or very similar QoS and CoS
characteristics.  In this case, nodes performing aggregation will
ensure that per-flow service requirements are achieved.
Paragraph (version B):
If bandwidth reservations are used, the sum of the reservations
should be the sum of all the individual reservations, in other words,
the reservations should not create an over subscription of bandwidth
reservation.  If maximum delay bounds are used the system should
ensure that the aggregate does not exceed the delay bounds of the
component flows.
Paragraph (version B):
DetNet encapsulation is a data plane mechanism that can be used to
aggregate traffic.  Encapsulation can either be in the same service
type or in a different service type see dn_svc_encaps for examples.
When an encapsulation is used the choice of reserving a maximum
resource level and then tracking the services in the aggregated
service or adjusting the aggregated resources as the services are
added is implementation and technology specific.
Paragraph (version B):
DetNet flows at edges must be able to handle rejection to an
aggregation group due to lack of resources as well as conditions
where general requirements are not satisfied.
Paragraph (version B):
IP aggreggation has both data plane and controller plane aspects.
For the dataplane flows may be aggregated for treatment based on
shared charateristice such as 5-tuple.  Alternatively, an IP
encapsualtion may be used to tunnel an aggregate number of DetNet
Flows between relay nodes.
Paragraph (version B):
MPLS aggragation similarly has data plane and controller plane
aspects.  In the case of MPLS flows are often tunneled in a
forwarding sublayer and reservation is associated with that MPLS
tunnel.
Paragraph (version A):
Specifics related to non-MPLS DetNet end station behavior are out
side the scope of this document.  For example, details on support for
DetNet IP data flows can be found in I-D.ietf-detnet-ip.  This
document is also useful for end stations that map IP flows to DetNet
flows.
Paragraph (version A):
As a general rule, DetNet MPLS domains are capable of forwarding any
DetNet MPLS flows and the DetNet domain does not mandate the end-
system or edge system encapsulation format.  Unless there is a proxy
of some form present, end-systems peer with similar end-systems using
the same application encapsulation format.  For example, as shown in
fig_es_connect, IP applications peer with IP applications and
Ethernet L2VPN applications peer with Ethernet L2VPN applications.
Paragraph (version A):
As shown in , MPLS nodes are interconnected by different sub-network
technologies, which may include point-to-point links.  Each of these
need to provide appropriate service to DetNet flows.  In some cases,
e.g., on dedicated point-to-point links or TDM technologies, all that
is required is for a DetNet node to appropriately queue its output
traffic.  In other cases, DetNet nodes will need to map DetNet flows
to the flow semantics (i.e., identifiers) and mechanisms used by an
underlying sub-network technology.  shows several examples of header
formats that can be used to carry DetNet MPLS flows over different
sub-network technologies.  L2 represent a generic layer-2
encapsulation that might be used on a point-to-point link.  TSN
represents the encapsulation used on an IEEE 802.1 TSN network, as
described in .  UDP/IP represents the encapsulation used on a DetNet
IP PSN, as described in .
Paragraph (version B):
As a general rule, DetNet domains are capable of forwarding any
DetNet flows and the DetNet domain does not mandate the end-system or
edge system encapsulation format.  Unless there is a proxy of some
form present, end-systems peer with similar end-systems using the
same application encapsulation format.  For example, as shown in
fig_es_connect, IP applications peer with IP applications and
Ethernet L2VPN applications peer with Ethernet L2VPN applications.
Paragraph (version B):
Any of the DetNet native service types may be transported by another
DetNet service.  MPLS nodes may interconnected by different sub-
network technologies, which may include point-to-point links.  Each
of these sub-network technologies need to provide appropriate service
to DetNet flows.  In some cases, e.g., on dedicated point-to-point
links or TDM technologies, all that is required is for a DetNet node
to appropriately queue its output traffic.  In other cases, DetNet
nodes will need to map DetNet flows to the flow semantics (i.e.,
identifiers) and mechanisms used by an underlying sub-network
technology.  fig_dn_mpls_sn_ex shows several examples of header
formats that can be used to carry DetNet MPLS flows over different
sub-network technologies.  L2 represent a generic layer-2
encapsulation that might be used on a point-to-point link.  TSN
represents the encapsulation used on an IEEE 802.1 TSN network, as
described in I-D.mpls-over-tsn.  UDP/IP represents the encapsulation
used on a DetNet IP PSN, as described in I-D.mpls-over-udp-ip.
Paragraph (version B):
In the case of MPLS Manage DetNet S-Label and F-Label allocation
and distribution, where the DetNet MPLS encapsulation is in use
see mpls_cp_considerations.
Paragraph (version B):
The ability to support DetNet flow aggregation.
Paragraph (version B):
Advertise static and dynamic node and link resources such as
capabilities and adjacencies to other network nodes (for dynamic
signaling approaches) or to network controllers (for centralized
approaches).
Paragraph (version B):
Scale to handle the number of DetNet flows expected in a domain
(which may require per-flow signaling or provisioning).
Paragraph (version B):
Provision flow identification information at each of the nodes
along the path.  Flow identification may differ depending on the
location in the network and the DetNet functionality (e.g. transit
node vs. relay node).
Paragraph (version B):
In the abstract, the results of either distributed signaling or
centralized provisioning are equivalent from a DetNet data plane
perspective - flows are instantiated, explicit routes are determined,
resources are reserved, and packets are forwarded through the domain
using the DetNet native data plane.
Paragraph (version A):
sec_dt_dp introduces the use of the IP "6-tuple" for flow
identification, and goes on to discuss how identified flows use
specific QoS mechanisms for flow-specific traffic treatment,
including path control and resource allocation.  XXX contains
detailed DetNet IP flow identification procedures.  Flow
identification will play an important role for the DetNet controller
plane.
Paragraph (version A):
XXX and YYY discuss the use of flow aggregation in DetNet.  Flow
aggregation can be accomplished using any of the 6-tuple fields
defined in XXX , using a DSCP identified traffic class or other
field.  It will be the responsibility of the DetNet controller plane
to be able to properly provision the use of these aggregation
mechanisms.  These requirements are included in control-management-
requirements.
Paragraph (version A):
Explicit routes are used to ensure that packets are routed through
the resources that have been reserved for them, and hence provide the
DetNet application with the required service.  A requirement for the
DetNet Controller Plane will be the ability to assign a particular
identified DetNet IP flow to a path through the DetNet domain that
has been assigned the required nodal resources to provide the
appropriate traffic treatment for the flow, and also to include
particular links as a part of the path that are able to support the
DetNet flow, for example by using IEEE 802.1 TSN links (as discussed
in XXX ).  Further considerations and requirements for the DetNet
Controller Plane are discussed in control-management-requirements.
Paragraph (version B):
Flow aggregation includes aggregation accomplished through the use of
hierarchical LSPs in MPLS and tunnels, in the case of IP, MPLS and
TSN, both of which aggregate multiple DetNet flows into a single new
DetNet flow.  It can also be grouping of IP flows that share 5-tuple
of 6-tuple attributes or flow identifiers at the DetNet layer.
Paragraph (version B):
Available resources are tracked through control plane or
management plane databases and distributed amongst controllers
or nodes that can manage resources.
Paragraph (version B):
When DetNet services are provisioned or requested one or more
paths meeting the requirements are selected and the resources
verified and if successful recorded in the databases.
Paragraph (version B):
The assignment of resources depends on the technology and it
includes assignment of specific links and coordination of the
queuing and other traffic management capabilities.
Paragraph (version B):
Depending on the specific technology the assigned resources are
updated in the databases preventing oversubscription.
Paragraph (version B):
Explicit routes are used to ensure that packets are routed through
the resources that have been reserved for them, and hence provide the
DetNet application with the required service.  A requirement for the
DetNet Controller Plane will be the ability to assign a particular
identified DetNet IP flow to a path through the DetNet domain that
has been assigned the required nodal resources.  This provides the
appropriate traffic treatment for the flow and also includes
particular links as a part of the path that are able to support the
DetNet flow.  For example, by using IEEE 802.1 TSN links (as
discussed in XXX ) DetNet parameters can be maintained.  Further
considerations and requirements for the DetNet Controller Plane are
discussed in control-management-requirements.
Paragraph (version A):
The path could be used a distributed control plane such as RFC2205
or RSVP-TE RFC3473 extended to support DetNet IP flows.
Paragraph (version B):
The path could use a distributed control plane such as RFC2205 or
RSVP-TE RFC3473 extended to support DetNet IP flows.
Paragraph (version A):
As discussed in sec_intro, this document does not specify the
mechanisms needed to eliminate contention loss or reduce jitter for
DetNet flows at the DetNet forwarding sub-layer.  The ability to
manage node and link resources to be able to provide these functions
will be a necessary part of the DetNet controller plane.  It will
also be necessary to be able to control the required queuing
mechanisms used to provide these functions along a flow's path
through the network.  See XXX and control-management-requirements for
further discussion of these requirements.
Paragraph (version A):
Some DetNet applications generate bidirectional traffic.  Although
this document discusses the DetNet IP data plane, MPLS definitions
RFC5654 are useful to illustrate terms such as associated
bidirectional flows and co-routed bidirectional flows.  MPLS defines
a point-to-point associated bidirectional LSP as consisting of two
unidirectional point-to-point LSPs, one from A to B and the other
from B to A, which are regarded as providing a single logical
bidirectional forwarding path.  This is analogous to standard IP
routing.  MPLS defines a point-to-point co-routed bidirectional LSP
as an associated bidirectional LSP which satisfies the additional
constraint that its two unidirectional component LSPs follow the same
path (in terms of both nodes and links) in both directions.  An
important property of co-routed bidirectional LSPs is that their
unidirectional component LSPs share fate.  In both types of
bidirectional LSPs, resource reservations may differ in each
direction.  The concepts of associated bidirectional flows and co-
routed bidirectional flows can also be applied to DetNet IP flows.
Paragraph (version B):
As discussed in sec_intro, this document does not specify the
mechanisms needed to eliminate packet contention, packet loss or
reduce jitter for DetNet flows at the DetNet forwarding sub-layer.
The ability to manage node and link resources to be able to provide
these functions is a necessary part of the DetNet controller plane.
It is also necessary to be able to control the required queuing
mechanisms used to provide these functions along a flow's path
through the network.  See XXX and control-management-requirements for
further discussion of these requirements.
Paragraph (version B):
DetNet applications typically generate bidirectional traffic.  IP and
MPLS typically treat each direction separately and do not force
interdependence of each direction.  MPLS has considered bidirectional
traffic requirements and the MPLS definitions from RFC5654 are useful
to illustrate terms such as associated bidirectional flows and co-
routed bidirectional flows.  MPLS defines a point-to-point associated
bidirectional LSP as consisting of two unidirectional point-to-point
LSPs, one from A to B and the other from B to A, which are regarded
as providing a single logical bidirectional forwarding path.  This is
analogous to standard IP routing.  MPLS defines a point-to-point co-
routed bidirectional LSP as an associated bidirectional LSP which
satisfies the additional constraint that its two unidirectional
component LSPs follow the same path (in terms of both nodes and
links) in both directions.  An important property of co-routed
bidirectional LSPs is that their unidirectional component LSPs share
fate.  In both types of bidirectional LSPs, resource reservations may
differ in each direction.  The concepts of associated bidirectional
flows and co-routed bidirectional flows can also be applied to DetNet
IP flows.
Paragraph (version A):
While the DetNet IP data plane must support bidirectional DetNet
flows, there are no special bidirectional features with respect to
the data plane other than the need for the two directions of a co-
routed bidirectional flow to take the same path.  That is to say that
bidirectional DetNet flows are solely represented at the management
and control plane levels, without specific support or knowledge
within the DetNet data plane.  Fate sharing and associated vs. co-
routed bidirectional flows can be managed at the control level.
Paragraph (version A):
Control and management mechanisms will need to support bidirectional
flows, but the specification of such mechanisms are out of scope of
this document.  An example control plane solution for MPLS can be
found in RFC7551.
Paragraph (version A):
Provision flow identification information at each of the nodes
along the path, and it may differ depending on the location in the
network and the DetNet functionality.
Paragraph (version A):
These requirements, as stated earlier, could be satisfied using
distributed control protocol signaling, centralized network
management provisioning mechanisms, or hybrid combinations of the
two, and could also make use of IPv6-based segment routing.
Paragraph (version A):
In the abstract, the results of either distributed signaling or
centralized provisioning are equivalent from a DetNet data plane
perspective - flows are instantiated, explicit routes are determined,
resources are reserved, and packets are forwarded through the domain
using the IP data plane.
Paragraph (version B):
While the DetNet IP data plane must support bidirectional DetNet
flows, there are no special bidirectional features with respect to
the data plane other than the need for the two directions of a co-
routed bidirectional flow to take the same path.  That is to say that
bidirectional DetNet flows are solely represented at the management
and control plane levels, without specific support or knowledge
within the DetNet data plane.  Fate sharing and associated or co-
routed bidirectional flows, can be managed at the control level.
Paragraph (version B):
DetNet's use of PREOF may increase the complexity of using co-routing
bidirectional flows, since if PREOF is used, then the replication
points in one direction would have to match the elimination points in
the other direction, and vice versa, and the optimal points for these
functions in one direction may not match the optimal points in the
other subsequent to the network and traffic constraints.
Paragraph (version B):
Control and management mechanisms need to support bidirectional
flows, but the specification of such mechanisms are out of scope of
this document.  An example control plane solution for MPLS can be
found in .  Related control plan mechanisms have been defined in
RFC3473 , RFC6387 and RFC7551.
Paragraph (version B):
sec_dt_dp discussed the use of the IP "6-tuple" for flow
identification, and goes on to discuss how identified flows use
specific QoS mechanisms for flow-specific traffic treatment,
including path control and resource allocation.  XXX contains
detailed DetNet IP flow identification procedures.  Flow
identification will play an important role for the DetNet controller
plane.
Paragraph (version B):
aggregation and ip-agg discuss the use of flow aggregation in DetNet.
Flow aggregation can be accomplished using any of the 6-tuple fields
defined in XXX , using a DSCP identified traffic class or other
field.  It will be the responsibility of the DetNet controller plane
to be able to properly provision the use of these aggregation
mechanisms.  These requirements are included in control-management-
requirements.
Paragraph (version A):
See mpls-control-management-requirements for further discussion of
these alternatives.
Paragraph (version A):
Much like other MPLS labels, there are a number of alternatives
available for DetNet S-Label and F-Label advertisement to an upstream
peer node.  These include distributed signaling protocols such as
RSVP-TE, centralized label distribution via a controller that manages
both the sender and the receiver using NETCONF/YANG, BGP, PCEP, etc.,
and hybrid combinations of the two.  The details of the controller
plane solution required for the label distribution and the management
of the label number space are out of scope of this document, but as
mentioned above, there are particular DetNet considerations and
requirements that are discussed in mpls-control-management-
requirements.
Paragraph (version B):
Much like other MPLS labels, there are a number of alternatives
available for DetNet S-Label and F-Label advertisement to an upstream
peer node.  These include distributed signaling protocols such as
RSVP-TE, centralized label distribution via a controller that manages
both the sender and the receiver using NETCONF/YANG, BGP, PCEP, etc.,
and hybrid combinations of the two.  The details of the controller
plane solution required for the label distribution and the management
of the label number space are out of scope of this document, but as
mentioned above, there are particular DetNet considerations and
requirements that are discussed in control-management-requirements.
Paragraph (version A):
through the network.  See mpls-control-management-requirements for
further discussion of these requirements.
Paragraph (version A):
There are still two bidirectional sections that need to be merged.
Paragraph (version A):
Some DetNet applications generate bidirectional traffic.  Using MPLS
definitions RFC5654 there are associated bidirectional flows, and co-
routed bidirectional flows.  MPLS defines a point-to-point associated
bidirectional LSP as consisting of two unidirectional point-to-point
LSPs, one from A to B and the other from B to A, which are regarded
as providing a single logical bidirectional forwarding path.  This
would be analogous of standard IP routing, or PWs running over two
reciprocal unidirection LSPs.  MPLS defines a point-to-point co-
routed bidirectional LSP as an associated bidirectional LSP which
satisfies the additional constraint that its two unidirectional
component LSPs follow the same path (in terms of both nodes and
links) in both directions.  An important property of co-routed
bidirectional LSPs is that their unidirectional component LSPs share
fate.  In both types of bidirectional LSPs, resource reservations may
differ in each direction.  The concepts of associated bidirectional
flows and co-routed bidirectional flows can be applied to DetNet
flows.
Paragraph (version A):
While the MPLS data plane must support bidirectional DetNet flows,
there are no special bidirectional features with respect to the data
plane other than the need for the two directions of a co-routed
bidirectional flow to take the same path.  Fate sharing and
associated vs co-routed bidirectional flows can be managed at the
control level.  Note that there is no stated requirement for
bidirectional DetNet flows to be supported using the same MPLS Labels
in each direction.
Paragraph (version A):
DetNet's use of PREOF may increase the complexity of using co-routing
bidirectional flows, since if PREOF is used, then the replication
points in one direction would have to match the elimination points in
the other direction, and vice versa, and the optimal points for these
functions in one direction may not match the optimal points in the
other.
Paragraph (version A):
Control and management mechanisms will need to support bidirectional
flows, but the specification of such mechanisms are out of scope of
this document.  Related control plan mechanisms have been defined in
RFC3473, RFC6387 and RFC7551.
Paragraph (version A):
This is further discussed in mpls-control-management-requirements.
Paragraph (version A):
XXX discusses the use of flow aggregation in DetNet.  It includes
flow aggregation accomplished through the use of hierarchical LSPs,
aggregating multiple DetNet flows into a single new DetNet flow, and
simple aggregation at the DetNet layer.  It will be the
responsibility of the DetNet controller plane to be able to properly
provision the use of these mechanisms.  These requirements are
included in the next section.
Paragraph (version A):
Instantiate DetNet flows in a DetNet domain (which may include
some or all of explicit path and PREOF replication and elimination
node determination, link bandwidth reservations, node buffer anad
other resource reservations, specification of required queuing
disciplines along the path, ability to manage bidirectional flows,
etc.) as needed for a flow.
Paragraph (version A):
Provision flow identification information at each of the nodes
along the path, and it may differ depending on the location in the
network and the DetNet functionality (e.g. transit node vs. relay
node).
Paragraph (version A):
In the abstract, the results of either distributed signaling or
centralized provisioning are equivalent from a DetNet data plane
perspective - flows are instantiated, explicit routes are determined,
resources are reserved, and packets are forwarded through the domain
using the MPLS data plane.
Paragraph (version B):
through the network.  See control-management-requirements for further
discussion of these requirements.
Paragraph (version A):
This document specifies the DetNet data plane operation for IP hosts
and routers that provide DetNet service to IP encapsulated data.  No
DetNet specific encapsulation is defined to support IP flows, instead
the existing IP and higher layer protocol header information is used
to support flow identification and DetNet service delivery.
Paragraph (version B):
This document specifies the DetNet data plane operation for IP hosts
and routers that provide DetNet service to IP encapsulated data.  No
DetNet specific encapsulation is defined to support IP flows, instead
the existing IP and higher layer protocol header information is used
to support flow identification and DetNet service delivery.  Common
data plane procedures and control information for all DetNet data
planes can be found in the I-D.ietf-detnet-framework.
Paragraph (version A):
This document describes how IP is used by DetNet nodes, i.e., hosts
and routers, identify DetNet flows and provide a DetNet service using
an IP data plane.  From a data plane perspective, an end-to-end IP
model is followed.  As mentioned above, existing IP and higher layer
protocol header information is used to support flow identification
and DetNet service delivery.
Paragraph (version A):
The DetNet IP dataplane uses "6-tuple" based flow identification,
Paragraph (version B):
This document describes how IP is used by DetNet nodes, i.e., hosts
and routers, identify DetNet flows and provide a DetNet service using
an IP data plane.  From a data plane perspective, an end-to-end IP
model is followed.  As mentioned above, existing IP and higher layer
protocol header information is used to support flow identification
and DetNet service delivery.  Common data plane procedures and
control information for all DetNet data planes can be found in the I-
D.ietf-detnet-framework.
Paragraph (version B):
The DetNet IP data plane uses "6-tuple" based flow identification,
Paragraph (version A):
Note: The operation of IEEE802.1 TSN end systems over DetNet enabled
IP networks is not described in this document.
Paragraph (version B):
Note: The operation of IEEE802.1 TSN end systems over DetNet enabled
IP networks is not described in this document.  TSN over MPLS is
discribed in I-D.ietf-detnet-tsn-over-mpls.
Paragraph (version A):
Implementations if this document MUST ensure that a DetNet flow
receives the traffic treatment that is provisioned for it via
configuration or the controller plane, e.g., via [YANG-REF-TBD].
General information on DetNet service can be found in I-D.ietf-
detnet-flow-information-model.  Typical mechanisms used to provide
different treatment to different flows includes the allocation of
system resources (such as queues and buffers) and provisioning or
related parameters (such as shaping, and policing).  Support can also
be provided via an underlying network technology such as MPLS I-
D.ietf-ip-over-mpls.  and IEEE802.1 TSN I-D.ietf-ip-over-tsn.  Other
than in the TSN case, the specific mechanisms used by a DetNet node
to ensure DetNet service delivery requirements are met for supported
DetNet flows is outside the scope of this document.
Paragraph (version B):
Implementations if this document MUST ensure that a DetNet flow
receives the traffic treatment that is provisioned for it via
configuration or the controller plane, e.g., via [YANG-REF-TBD].
General information on DetNet service can be found in I-D.ietf-
detnet-flow-information-model.  Typical mechanisms used to provide
different treatment to different flows includes the allocation of
system resources (such as queues and buffers) and provisioning or
related parameters (such as shaping, and policing).  Support can also
be provided via an underlying network technology such as MPLS I-
D.ietf-detnet-ip-over-mpls.  and IEEE802.1 TSN I-D.ietf-ip-over-tsn.
Other than in the TSN case, the specific mechanisms used by a DetNet
node to ensure DetNet service delivery requirements are met for
supported DetNet flows is outside the scope of this document.