Original text:
DetNet Data Plane: IEEE 802.1 Time Sensitive Networking over MPLS <del> draft-ietf-detnet-tsn-vpn-over-mpls-00 </del>  Abstract This document specifies the Deterministic Networking data plane when <del> TSN networks interconnected over an MPLS Packet Switched Networks. </del>  1. <del> [Editor's note: Introduction to be made specific to TSN over DetNet scenario.  Do we intend to cover both TSN over DetNet IP and TSN over DetNet MPLS?  Or this document is limited to MPLS scenarios?]. </del>  <del> 2. </del>  <del> [Editor's note: text to be review what is really needed here.]. </del>  2.1. <del> This document uses the terminology established in the DetNet architecture I-D.ietf-detnet-architecture, and the reader is assumed to be familiar with that document and its terminology. The following terminology is introduced in this document: A Detnet "forwarding" label that identifies the LSP used to forward a DetNet flow across an MPLS PSN, e.g., a hop-by-hop label used between label switching routers (LSR). A DetNet "service" label that is used between DetNet nodes that implement also the DetNet service sub-layer functions.  An S-Label is also used to identify a DetNet flow at DetNet service sub- layer. A DetNet Control Word (d-CW) is used for sequencing and identifying duplicate packets of a DetNet flow at the DetNet service sub-layer. </del>  2.2. DetNet Flow. <del> DetNet Inter-Working Function. </del>  Layer 2. Time-Sensitive Network. <del> 3. </del>  The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 14 RFC2119 RFC8174 when, and only when, they appear in all capitals, as shown here. <del> 4. [Author's note: review required on his part.] </del>  fig_tsn_mpls_detnet shows IEEE 802.1 TSN end stations operating over a TSN aware DetNet service running over an MPLS network.  DetNet Edge encapsulation.  These are functionally similar to pseudowire (PW) Terminating Provider Edge (T-PE) nodes which use MPLS-TE LSPs.  In this example they understand and support IEEE 802.1 TSN and are able <del> to map TSN flows into DetNet flows.  The specific of this operation </del>  are discussed later in this document. <del> Native TSN flow and DetNet MPLS flow differ not only by the additional MPLS specific encapsulation, but DetNet MPLS flows have on each DetNet node an associated DetNet specific data structure, what defines flow related characteristics and required forwarding functions.  In this example, edge Nodes provide a service proxy function that "associates" the DetNet flows and native flows at the edge of the DetNet domain.  This ensures that the DN Flow is properly served at the Edge node (and inside the domain). </del>  fig_8021_detnet illustrates how DetNet can provide services for IEEE <del> 802.1TSN end systems, CE1 and CE2, over a DetNet enabled MPLS </del>  network.  Edge nodes, E1 and E2, insert and remove required DetNet data plane encapsulation.  The 'X' in the edge nodes and relay node, R1, represent a potential DetNet compound flow packet replication and elimination point.  This conceptually parallels L2VPN services, and could leverage existing related solutions as discussed below. <del> 5. [Editor's note: Needs clean up, what is relevant for TSN over DetNet scenarios.]. This section provides informative considerations related to providing DetNet service to flows which are identified based on their header information.  At a high level, the following are provided on a per flow basis: Use of allocated resources (queuing, policing, shaping) to ensure that the congestion-related loss and latency/jitter requirements of a DetNet flow are met. Use of a specific path for a flow.  This limits misordering and bounds latency. Which in the case of this document primarily relates to replication and elimination.  Changing the explicit path after a failure is detected in order to restore delivery of the required DetNet service characteristics is also possible.  Path changes, even in the case of failure recovery, can lead to the out of order delivery of data. Generally, distributing packets of the same DetNet flow over multiple paths is not recommended.  Such load sharing, e.g., via ECMP or UCMP, impacts ordering and possibly jitter. For example, to support identification of misbehaving flows. </del>  <del> For example, increase counters. </del>  <del> For example, unexpected loss. </del>  <del> The DetNet data plane also allows for the aggregation of DetNet flows, e.g., via MPLS hierarchical LSPs, to improved scaling.  When DetNet flows are aggregated, transit nodes provide service to the aggregate and not on a per-DetNet flow basis.  In this case, nodes performing aggregation will ensure that per-flow service requirements are achieved. </del>  5.1. <del> Data-flows requiring DetNet service are generated and terminated on end-systems.  Encapsulation depends on application and its preferences.  In a DetNet MPLS domain the DN functions use the d-CWs, S-Labels and F-Labels to provide DetNet services.  However, an application may exchange further flow related parameters (e.g., time- stamp), which are not provided by DN functions. Specifics related to non-MPLS DetNet end station behavior are out side the scope of this document.  For example, details on support for DetNet IP data flows can be found in I-D.ietf-detnet-dp-sol-ip.  This document is also useful for end stations that map IP flows to DetNet flows. As a general rule, DetNet MPLS domains are capable of forwarding any DetNet MPLS flows and the DetNet domain does not mandate the end- system or edge system encapsulation format.  Unless there is a proxy of some form present, end-systems peer with similar end-systems using the same application encapsulation format.  For example, as shown in fig_es_connect, IP applications peer with IP applications and Ethernet L2VPN applications peer with Ethernet L2VPN applications. 6. [Editor's note: Needs clean up.  Text should focus on Edge node related topics.]. 6.1. To carry DetNet over MPLS the following is required: A method of identifying the MPLS payload type. A method of identifying the DetNet flow group to the processing element. A method of distinguishing DetNet OAM packets from DetNet data packets. A method of carrying the DetNet sequence number. A suitable LSP to deliver the packet to the egress PE. A method of carrying queuing and forwarding indication. In this design an MPLS service label (the S-Label), similar to a pseudowire (PW) label RFC3985, is used to identify both the DetNet flow identity and the payload MPLS payload type satisfying (1) and (2) in the list above.  OAM traffic discrimination happens through the use of the Associated Channel method described in RFC4385.  The DetNet sequence number is carried in the DetNet Control word which carries the Data/OAM discriminator.  To simplify implementation and to maximize interoperability two sequence number sizes are supported: a 16 bit sequence number and a 28 bit sequence number.  The 16 bit sequence number is needed to support some types of legacy clients. The 28 bit sequence number is used in situations where it is necessary ensure that in high speed networks the sequence number space does not wrap whilst packets are in flight. </del>  The LSP used to forward the DetNet packet may be of any type (MPLS- LDP, MPLS-TE, MPLS-TP RFC5921, or MPLS-SR I-D.ietf-spring-segment- routing-mpls).  The LSP (F-Label) label and/or the S-Label may be used to indicate the queue processing as well as the forwarding <del> parameters.  Note that the possible use of Penultimate Hop Popping (PHP) means that the only label in a received label stack may be the S-Label. 6.2. 6.2.1. An edge node is resposable for matching ingress packets to the service they require and encapsulating them accordingly.An edge node may participate in the packet replication and duplication elimination. The DetNet-aware forwarder selects the egress DetNet member flow segment based on the flow identification.  The mapping of ingress DetNet member flow segment to egress DetNet member flow segment may be statically or dynamically configured.  Additionally the DetNet- aware forwarder does duplicate frame elimination based on the flow identification and the sequence number combination.  The packet replication is also done within the DetNet-aware forwarder.  During elimination and the replication process the sequence number of the DetNet member flow MUST be preserved and copied to the egress DetNet member flow. The internal design of a relay node is out of scope of this document. However the reader's attention is drawn to the need to make any PREOF state available to the packet processor(s) dealing with packets to which the PREOF functions must be applied, and to maintain that state is such as way that it is available to the packet processor operation on the next packet in the DetNet flow (which may be a duplicate, a late packet, or the next packet in sequence. [Editor's note: I think the rest of this section belongs in a new "802.1 TSN (island Interconnect) over DetNet MPLS" section.] This may be done in the DetNet layer, or where the native service processing (NSP) RFC3985 is IEEE 802.1CB IEEE8021CB capable, the packet replication and duplicate elimination MAY entirely be done in the NSP, bypassing the DetNet flow encapsulation and logic entirely. This enables operating over unmodified implementations and deployments.  The NSP approach works only between edge nodes and cannot make use of relay nodes. The NSP approach is useful end to end tunnel and for for "island interconnect" scenarios.  However, when there is a need to do PREOF in a middle of the network, such plain edge to edge operation is not sufficient. The extended forwarder MAY copy the sequencing information from the native DetNet packet into the DetNet sequence number field and vice versa.  If there is no existing sequencing information available in the native packet or the forwarder chose not to copy it from the native packet, then the extended forwarder MUST maintain a sequence number counter for each DetNet flow (indexed by the DetNet flow identification). 6.2.2. [Editor's NOTE: review and simplify this section if possible.] </del>  <del> The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1 Working Group have defined (and are defining) a number of amendments to IEEE8021Q that provide zero congestion loss and bounded latency in bridged networks.  IEEE8021CB defines packet replication and elimination functions that should prove both compatible with and useful to, DetNet networks. As is the case for DetNet, a Layer 2 network node such as a bridge may need to identify the specific DetNet flow to which a packet belongs in order to provide the TSN/DetNet QoS for that packet.  It also will likely need a CoS marking, such as the priority field of an IEEE Std 802.1Q VLAN tag, to give the packet proper service. Although the flow identification methods described in IEEE8021CB are flexible, and in fact, include IP 5-tuple identification methods, the baseline TSN standards assume that every Ethernet frame belonging to a TSN stream (i.e.  DetNet flow) carries a multicast destination MAC address that is unique to that flow within the bridged network over which it is carried.  Furthermore, IEEE8021CB describes three methods by which a packet sequence number can be encoded in an Ethernet frame. Ensuring that the proper Ethernet VLAN tag priority and destination MAC address are used on a DetNet/TSN packet may require further clarification of the customary L2/L3 transformations carried out by routers and edge label switches.  Edge nodes may also have to move sequence number fields among Layer 2, PW, and IP encapsulations. </del>  <del> 7. </del>  <del> [Editor's note: requires considerations related to TSN over DetNet.]. </del>  <del> 8. </del>  The security considerations of DetNet in general are discussed in I- <del> D.ietf-detnet-architecture and I-D.sdt-detnet-security.  Other security considerations will be added in a future version of this draft. </del>  <del> 9. </del>  This document makes no IANA requests.

Reviewer comments:
Major clean up and simplification as data plane drafts can be referred for MPLS related details. TSN over DetNet MPLS follows the concept of RFC3985. Description focus on Edge Nodes procedures and functions. Text covers the Edge Nodes components: (1) TSN functions (in NSP block); (2) DetNet Service Proxy; and (3) DetNet sub-layers.
Approving from teelcon review

Revised text:
DetNet Data Plane: IEEE 802.1 Time Sensitive Networking over MPLS  <ins> draft-ietf-detnet-tsn-vpn-over-mpls-01 </ins> Abstract This document specifies the Deterministic Networking data plane when  <ins> TSN networks are interconnected over a DetNet MPLS Network. </ins> 1.  <ins> The Time-Sensitive Networking Task Group (TSN TG) within IEEE 802.1 Working Group deals with deterministic services through IEEE 802 networks.  Deterministic Networking (DetNet) defined by IETF is a service that can be offered by a L3 network to DetNet flows.  General background and concepts of DetNet can be found in I-D.ietf-detnet- architecture. </ins>  <ins> This document specifies the use of a DetNet MPLS network to interconnect TSN nodes/network segments.  DetNet MPLS data plane is defined in I-D.ietf-detnet-mpls. </ins>  <ins> 2. </ins> 2.1.  <ins> This document uses the terminology and concepts established in the DetNet architecture I-D.ietf-detnet-architecture and I-D.ietf-detnet- data-plane-framework, and I-D.ietf-detnet-mpls.  The reader is assumed to be familiar with these documents and their terminology. </ins> 2.2. DetNet Flow.  <ins> Frame Replication and Elimination for Redundancy (TSN function). </ins> Layer 2. Time-Sensitive Network.  <ins> 2.3. </ins> The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and 14 RFC2119 RFC8174 when, and only when, they appear in all capitals, as shown here.  <ins> 3. </ins> fig_tsn_mpls_detnet shows IEEE 802.1 TSN end stations operating over a TSN aware DetNet service running over an MPLS network.  DetNet Edge encapsulation.  These are functionally similar to pseudowire (PW) Terminating Provider Edge (T-PE) nodes which use MPLS-TE LSPs.  In this example they understand and support IEEE 802.1 TSN and are able  <ins> to map TSN Streams into DetNet flows.  The specific of this operation </ins> are discussed later in this document.  <ins> In this example, edge nodes provide a service proxy function that "associates" the DetNet flows and native flows (i.e., TSN Streams) at the edge of the DetNet domain.  This ensures that the DN Flow is properly served at the Edge node (and inside the domain). </ins> fig_8021_detnet illustrates how DetNet can provide services for IEEE  <ins> 802.1 TSN end systems, CE1 and CE2, over a DetNet enabled MPLS </ins> network.  Edge nodes, E1 and E2, insert and remove required DetNet data plane encapsulation.  The 'X' in the edge nodes and relay node, R1, represent a potential DetNet compound flow packet replication and elimination point.  This conceptually parallels L2VPN services, and could leverage existing related solutions as discussed below.  <ins> 4. 4.1. The basic approach defined in I-D.ietf-detnet-mpls supports the DetNet service sub-layer based on existing pseudowire (PW) encapsulations and mechanisms, and supports the DetNet forwarding sub-layer based on existing MPLS Traffic Engineering encapsulations and mechanisms. A node operating on a DetNet flow in the Detnet service sub-layer, i.e.  a node processing a DetNet packet which has the S-Label as top of stack uses the local context associated with that S-Label, for example a received F-Label, to determine what local DetNet operation(s) are applied to that packet.  An S-Label may be unique when taken from the platform label space RFC3031, which would enable correct DetNet flow identification regardless of which input interface or LSP the packet arrives on.  The service sub-layer functions (i.e., PREOF) use a DetNet control word (d-CW). The DetNet MPLS data plane builds on MPLS Traffic Engineering encapsulations and mechanisms to provide a forwarding sub-layer that is responsible for providing resource allocation and explicit routes. The forwarding sub-layer is supported by one or more forwarding labels (F-Labels). DetNet edge/relay nodes are DetNet service sub-layer aware, understand the particular needs of DetNet flows and provide both DetNet service and forwarding sub-layer functions.  They add, remove and process d-CWs, S-Labels and F-labels as needed.  MPLS enabled DetNet nodes can enhance the reliability of delivery by enabling the replication of packets where multiple copies, possibly over multiple paths, are forwarded through the DetNet domain.  They can also eliminate surplus previously replicated copies of DetNet packets. MPLS (DetNet) nodes also include DetNet forwarding sub-layer functions, support for notably explicit routes, and resources allocation to eliminate (or reduce) congestion loss and jitter. DetNet transit nodes reside wholly within a DetNet domain, and also provide DetNet forwarding sub-layer functions in accordance with the performance required by a DetNet flow carried over an LSP.  Unlike other DetNet node types, transit nodes provide no service sub-layer processing. 4.2. The basic encapsulation approach is to treat a TSN Stream as an app- flow from the DetNet MPLS perspective.  The corresponding example shown in fig_tsn_mpls_ex. In the figure, "Application" indicates the application payload carried by the TSN network.  "MPLS App-Flow" indicates that the TSN Stream is the payload from the perspective of the DetNet MPLS data plane defined in I-D.ietf-detnet-mpls.  A single DetNet MPLS flow can aggregate multiple TSN Streams. </ins>  <ins> 5. Description of Edge Nodes procedures and functions for TSN over DetNet MPLS scenario follows the concept of RFC3985 and covers the Edge Nodes components shown on fig_tsn_mpls_detnet.  In this section the following procedures of DetNet Edge Nodes are described: TSN related (tom_tsn_proc) </ins>  <ins> DetNet Service Proxy (tom_svc_prx_proc) </ins>  <ins> DetNet service and forwarding sub-layer (tom_dn_sub_proc) </ins> 5.1.  <ins> The Time-Sensitive Networking (TSN) Task Group of the IEEE 802.1 Working Group have defined (and are defining) a number of amendments to IEEE8021Q that provide zero congestion loss and bounded latency in bridged networks.  IEEE8021CB defines packet replication and elimination functions for a TSN network. TSN specific functions are executed on the data received by the PE from the CE before presentation to the DetNet PW for transmission across the DetNet domain, or on the data received from a DetNet PW by a PE before it is output on the Attachment Circuit (AC). TSN specific function(s) of Edge Nodes (T-PE) are belonging to the native service processing (NSP) RFC3985 block.  This is similar to other functionalities being defined by standard bodies other than the IETF (for example in case of Ethernet: stripping, overwriting or adding VLAN tags, etc.).  Depending on the TSN role of the Edge Node in the end-to-end TSN service selected TSN functions must be supported. Implementations of this document SHALL use management and control information to ensure TSN specific functions of the Edge Node according to the expectations of the connected TSN network. 5.2. The Service Proxy function maps between App-flows and DetNet flows. It MUST support the TSN Stream identification functions and the related managed objects as defined in IEEE8021CB and IEEEP8021CBdb to recognize the App-flow related packets. Implementations of this document SHALL use management and control information to map a TSN Stream to a DetNet flow.  N:1 mapping (aggregating TSN Streams in a single DetNet flow) SHALL be supported. The management or control function that provisions flow mapping SHALL ensure that adequate resources are allocated and configured to provide proper service requirements of the mapped flows. Due to the (intentional) similarities of the DetNet PREOF and TSN FRER functions service protection function interworking is possible between the TSN and the DetNet domains.  Such service protection interworking scenarios MAY require to copy sequence number fields from TSN (L2) to PW (MPLS) encapsulation.  However, such interworking is out-of-scope in this document and left for further study. 5.3. In the design of I-D.ietf-detnet-mpls an MPLS service label (the S-Label), similar to a pseudowire (PW) label RFC3985, is used to identify both the DetNet flow identity and the payload MPLS payload type.  The DetNet sequence number is carried in the DetNet Control word (d-CW) which carries the Data/OAM discriminator as well.  In I- D.ietf-detnet-mpls two sequence number sizes are supported: a 16 bit sequence number and a 28 bit sequence number. PREOF functions and the provided service recovery is available only within the DetNet domain as the DetNet flow-ID and the DetNet sequence number are not valid outside the DetNet network.  MPLS (DetNet) Edge node terminates all related information elements encoded in the MPLS labels. </ins> The LSP used to forward the DetNet packet may be of any type (MPLS- LDP, MPLS-TE, MPLS-TP RFC5921, or MPLS-SR I-D.ietf-spring-segment- routing-mpls).  The LSP (F-Label) label and/or the S-Label may be used to indicate the queue processing as well as the forwarding  <ins> parameters. </ins>  <ins> For further details see I-D.ietf-detnet-mpls. </ins>  <ins> 6. </ins>  <ins> [Editor's note: This section covers management/control plane related implications of creation, mapping, removal of DetNet flow IDs, their related parameters and, when needed, the configuration of PREOF.] TSN Stream(s) to DetNet flow mapping related information are required only for the service proxy function of MPLS (DetNet) Edge nodes. From the Data Plane perspective there is no practical difference based on the origin of flow mapping related information (management plane or control plane). MPLS DetNet Edge nodes are member of both the DetNet domain and the connected TSN network.  From the TSN network perspective the MPLS (DetNet) Edge node has a "TSN relay node" role, so TSN specific management and control plane functionalities must be implemented. There are many similarities in the management plane techniques used in DetNet and TSN, but that is not the case for the control plane protocols.  For example, RSVP-TE and MSRP behaves differently. Therefore management and control plane design is an important aspect of scenarios, where mapping between DetNet and TSN is required. In order to use a DetNet network to interconnect TSN segments, TSN specific information must be converted to DetNet domain specific ones.  TSN Stream ID(s) and stream(s) related parameters/requirements must be converted to a DetNet flow-ID and flow related parameters/ requirements.  Note that, as the DetNet network is just a portion of the end2end TSN path (i.e., single hop from Ethernet perspective), some parameters (e.g., delay) may differ significantly.  Other parameters (like bandwidth) also may have to be tuned due to the MPLS encapsulation used within the DetNet network. In some case it may be challenging to determine some egress node related information.  For example, it may be not trivial to locate the egress point/interface of a TSN Streams with a multicast destination MAC address.  Such scenarios may require interaction between control and management plane functions and between DetNet and TSN domains. Mapping between DetNet flow identifiers and TSN Stream identifiers, if not provided explicitly, can be done by the service proxy function of an MPLS (DetNet) Edge node locally based on information provided for configuration of the TSN Stream identification functions (e.g., Mask-and-Match Stream identification). Triggering the setup/modification of a DetNet flow in the DetNet network is an example where management and/or control plane interactions are required between the DetNet and the TSN network. Configuration of TSN specific functions (e.g., FRER) inside the TSN network is a TSN domain specific decision and may not be visible in the DetNet domain.  Service protection interworking scenarios are left for further study. </ins>  <ins> 7. </ins> The security considerations of DetNet in general are discussed in I-  <ins> D.ietf-detnet-architecture and I-D.ietf-detnet-security.  DetNet MPLS data plane specific considerations are summarized in I-D.ietf-detnet- mpls.  Encryption may provided within the TSN network using MACSec IEEE802.1AE-2018. </ins>  <ins> 8. </ins> This document makes no IANA requests.