Patent Description:
In distributed computing systems that use networks, especially those that use cloud or multi-cloud environments, multicast communications between applications and services implemented in software, as opposed to internetworking elements like routers and switches, often is needed. Currently, such one-to-many communication is implemented, if at all, at the application layer. Application programs and/or services are programmed to send repeated unicast application messages from a single source to multiple different recipients. This approach is highly inefficient with respect to use of network resources and bandwidth. It also introduces unnecessary traffic into packet-switched networks, which affects overall device and network performance.

When applications are implemented as services or microservices that have many connections, logically forming a mesh network of services, these problems become acute.

Thus, there is a need to efficiently process multicast traffic for applications. Examples include voice and video applications.

<CIT> describes, according to its abstract, various systems and methods for performing bit indexed explicit replication (BIER). For example, one method involves receiving a packet at a node. The packet includes a bit string. The node selects forwarding information based on a flow value associated with the packet. The forwarding information includes a forwarding bit mask. The node then forwards the packet based on the bit string and the forwarding information.

<CIT> describes, according to its abstract, methods and network devices for replication and switching of Internet Protocol (IP) packets in professional media networks. In one embodiment, a method includes encapsulating a unicast IP packet with a packet bit array and forwarding the encapsulated packet via a replication fabric within a network. In this embodiment, each receiver of a plurality of receivers reachable via the replication fabric is represented by a relative bit position in the packet bit array, a respective IP address is assigned to each receiver of the plurality of receivers, and the replication fabric is adapted to store disposition information mapping a relative bit position representing one or more of the plurality of receivers to IP addresses assigned to the one or more of the plurality of receivers. An embodiment of a network device includes a processor operably coupled to a network interface and adapted to perform steps of the method.

<NPL>) explains that, according to its abstract, currently deployed backbone/metro networks rarely supports dynamically configurable multicasting. One of the major reasons is the high complexity and low scalability of current multicasting solutions. Alternative multicasting solutions are emerging based on Software Defined Networking (SDN) using a central controller simplifies the multicast tree management but still requires the explicit configuration of all intermediate nodes.

<NPL>) describes, according to its abstract, Elmo, a system that addresses the multicast scalability problem in multi-tenant data centers. Elmo scales network multicast by taking advantage of emerging programmable switches and the unique characteristics of data-center networks; specifically, the symmetric topology and short paths in a data center. Elmo encodes multicast group information inside packets themselves, reducing the need to store the same information in network switches. In a three-tier data-center topology with <NUM> hosts, Elmo supports a million multicast groups using a <NUM>-byte packet header, requiring as few as <NUM> multicast group-table entries on average in leaf switches, with a traffic overhead as low as <NUM>% over ideal multicast.

An invention is defined in the appended claims.

In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present embodiments. It will be apparent, however, that the present embodiments may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present embodiments. Embodiments are described in sections below according to the following outline:.

This disclosure provides a digital data processing system and computer-implemented method that is capable of introducing the efficiency of network-layer multicast-like packet delivery within a service mesh. "Service mesh," in this context, refers to a plurality of services or microservices, which may comprise or be components of application programs, that communicate with one another. When each of the services communicates with all other services, a full service mesh may be formed, but is not required in all embodiments. Microservices may support applications, such as voice and video. Furthermore, embodiments use the Bit Index Explicit Replication (BIER) protocol and the iOAM (In-situ Operations, Administration, and Maintenance) protocol to enable multicast-like packet delivery in a service mesh, while also providing the means to deliver in-band definition of forwarding policies. BIER is defined in part in Request for Comments (RFC) <NUM>, <NUM> and RFC <NUM> of the Intemet Society and managed by the Intemet Engineering Task Force.

This disclosure assumes familiarity with and knowledge of the conventional use of BIER and iOAM and focuses on new and unexpected applications of the protocols to a different computing domain. These protocols have not previously been used in a service mesh to enable multicast-like packet delivery while allowing in-band policy selection. BIER headers include an entropy field that is used conventionally to provide simple forwarding adjustments at a replication node. For example, it can be used to color or tag traffic such that traffic tagged with the same entropy value is sent on the same path. However, it has not been used previously to transmit policy definitions, which represents a departure from all previously defined purposes of the header.

In an embodiment, BIER and iOAM are used to enable multicast-like packet delivery within a service mesh, while also providing the means to deliver in-band definition of forwarding policies. Additionally, an iOAM-based mechanism triggers dynamic policy enforcement on BIER replicators. In an embodiment, BIER bitmask traffic distribution is adapted to service mesh deployments. In embodiment, Sidecar functionality is updated to incorporate BIER replicator capabilities. In an embodiment, protocol-independent multicast-like packet delivery across cloud platforms is provided in a manner that is independent of the transport layer. In-band policy delivery using iOAM to replicator functionality within a service mesh is provided. A symbolic policy definition language can carry policy information, ranging from a policy identifier for already defined policies up to complete policy specifications. While certain embodiments are described for use with iOAM, in other embodiments useful for understanding the invention, other metadata-enabled protocols such as SRv6 or NSH may be used for policy delivery in networks other than service mesh networks.

Embodiments have the benefit of not requiring direct support in the network of multicast protocols such as PIM (Protocol Independent Multicast, RFC <NUM>), mLDP (Multicast Label Distribution Protocol), or RSVP-TE/P2MP (Resource Reservation Protocol - Traffic Engineering, RFC <NUM>, Point-to-Multipoint RFC <NUM>).

The claims <NUM>, <NUM> and <NUM> define respectively a method, a computer system and one or more non-transitory computer-readable storage media according to the invention. The dependent claims define various embodiments.

<FIG> is a two-part diagram that illustrates a service mesh. View (A) is a representation of an example service mesh at Layer <NUM> of the OSI network reference model, and view (B) is an implementation of the same service mesh using internetworking devices and software at Layer <NUM> to Layer <NUM>. View (A) represents a logical view of how micro-services may interact, but it does not represent the actual implementation in the network to support communicating between the micro-services.

Referring first to view (A), in an embodiment, an example service mesh <NUM> comprises a plurality of micro-services MS1, MS2, MS3, MS4, MS5, MS6, MS7 that are logically coupled by paths indicated by lines. Each of the micro-services MS1, MS2, MS3, MS4, MS5, MS6, MS7 may comprise a separate computer program, process or other software element. One or more of the micro-services MS1, MS2, MS3, MS4, MS5, MS6, MS7 may relate to the same broader application program and may be instantiated or launched by that program. One of the micro-services such as MS3 communicates with other micro-services by repetitive unicast packets. In an embodiment, multicast-like packets are used.

Turning now to view (B), in an embodiment, the service mesh <NUM> of view (A) may be implemented in a network using one or more replicator nodes <NUM>, <NUM> that communicate to one or more receiver nodes <NUM> over paths indicated by arrows. Each replicator <NUM>, <NUM> may store policy in memory to govern forwarding operations of the replicator. Both service mesh <NUM> and implementation <NUM> may be hosted using routers, switches or other internetworking elements in a public cloud infrastructure, private cloud infrastructure or multi-cloud infrastructure. Implementation in a non-cloud enterprise network or campus network also is possible.

In an embodiment, a particular micro-service such as MS3 may transmit an iOAM message comprising an iOAM policy header field. In one embodiment, the iOAM policy header field may carry a policy identifier that specifies a policy that is already defined and installed on a particular replicator <NUM>, <NUM>. The effect of such a message is to request the replicator <NUM>, <NUM> that receives the message to load and use the pre-defined policy for forwarding messages originating from the micro-service MS3 and directed to one or more of the receivers <NUM>.

Alternatively, the iOAM policy header field may expressly specify policy characteristics for policy enforcement on a particular replicator <NUM>, <NUM>. The effect of such a message is to request the replicator <NUM>, <NUM> that receives the message to parse specific policy instructions that are contained in the header field, and to use the parsed policy instructions for forwarding messages originating from the micro-service MS3 and directed to one or more of the receivers <NUM>.

Embodiments may be used to facilitate multicast-like traffic distribution for micro-services in several scenarios. First, one micro-service may operate as a source, with some or all of the remaining micro-services as receivers. Second, multicast traffic may arrive from the outside the service mesh, yet inside the control domain of a cloud computing facility, and needs to be handled by the micro-services within the mesh. Third, a source outside the micro-service mesh and outside the cloud control domain may have micro-services as receivers, such as for management operations.

<FIG> illustrates a combination of BIER and iOAM operations executing in an example service mesh. In the example of <FIG>, a service mesh <NUM> with multicast service messaging comprises a data source <NUM>, which may be hosted within the service mesh <NUM> or outside it. The data source <NUM> may comprise a micro-service, an application program, or any other software element that is programmed to transmit messages to any of a plurality of receivers <NUM>, <NUM>, <NUM>. Both data source <NUM> and receivers <NUM>, <NUM>, <NUM> may be implemented as individual computers, programs, other software elements, processes or applications.

Service mesh <NUM> further comprises a BIER controller <NUM> that is communicatively coupled to one or more BIER replicator nodes <NUM>, <NUM>. The BIER replicator nodes <NUM>, <NUM> are coupled to one or more BIER receivers <NUM>, <NUM>, <NUM> and the receivers typically are within the service mesh <NUM>. Each BIER receiver node <NUM>, <NUM>, <NUM> is uniquely associated with specific receivers among the one or more of the receivers <NUM>, <NUM>, <NUM> for purposes of local communication. For example, BIER receiver node <NUM> manages receiver <NUM>, BIER receiver node <NUM> manages receiver <NUM>, and BIER receiver node <NUM> manages receiver <NUM>. A particular receiver <NUM>, <NUM>, <NUM> is associated with and managed by only one BIER receiver node <NUM>, <NUM>, <NUM>. However, a particular BIER receiver node <NUM>, <NUM>, <NUM> may manage a large number or group of receivers.

In one embodiment, in which containerization software frameworks are used to manage execution of micro-services, each of the BIER controller <NUM>, BIER replicator nodes <NUM>, <NUM> and the one or more BIER receivers <NUM>, <NUM>, <NUM> may be implemented using shadow containers that execute in association with main containers that manage the micro-service, in cooperation with a virtualized container framework. For example, proxy containers or extra containers configured to manage aspects of the data plane and control plane traffic may be coupled to or associated with other containers in which micro-services execute. In an embodiment, each of the data source <NUM> and the receivers <NUM>, <NUM>, <NUM> executes in a different virtualized container. Examples of virtualized containerization frameworks include DOCKER, APACHE MESOS, RKT, and GARDEN.

Elements in service mesh <NUM> are logically and/or physically coupled on control and data plane paths that are indicated by arrows and defined in legend <NUM>. Paths indicated by different line styles indicate: traffic replication from the source <NUM> via the replicator nodes <NUM>, <NUM> to one or more receivers <NUM>, <NUM>, <NUM>; BIER control plane traffic between BIER-enabled network elements such as receiver nodes <NUM>, <NUM>, <NUM>, and the BIER controller <NUM>, which may include join messages that the receiver nodes received from the receivers <NUM>, <NUM>, <NUM>; and transmission of such multicast information that has been gathered from the receivers <NUM>, <NUM>, <NUM>, such as group membership or join information, to the replicator nodes <NUM>, <NUM> and the source <NUM>, from the BIER controller <NUM>.

In an embodiment, iOAM messages carried with substantive micro-service messages will trigger policy enforcement on BIER replicator nodes <NUM>, <NUM>. This approach recognizes that in-band policy enforcement on a per-flow basis may be important to a dynamic, on-demand and ever-changing container environment. Statically defined policies on a replicator node <NUM>, <NUM> may not change fast enough to cope with the dynamic behavior of container environments. However, in an embodiment, a modification of iOAM supports transmission of a policy definition to a replicator node <NUM>, <NUM>. The replicator node <NUM>, <NUM> then uses the policy definition to enhance BIER forwarding.

Example operations performed by BIER components are shown in <FIG> with numbers <NUM>, <NUM> and <NUM>, and modified iOAM operations are denoted A, B and C. The BIER operations comprise:.

Operation <NUM>. The source <NUM> prepares data for transmission to two or more receivers <NUM>, <NUM>, <NUM>. For example, assume that the source <NUM> sends a multicast stream for multicast group <NUM>. <NUM> to replicator node <NUM> and sends a multicast stream for multicast group <NUM>. <NUM> to replicator node <NUM>. Note that a single source <NUM> may initiate multiple multicast streams directed to different multicast groups and may direct them to different replicator nodes <NUM>, <NUM>. The source <NUM> queries the BIER controller <NUM> to obtain addresses or other forwarding data for one or more of the replicator nodes <NUM>, <NUM> that are capable of forwarding to receiver nodes <NUM>, <NUM>, <NUM> that can reach receivers <NUM>, <NUM>, <NUM>. The specific manner by which the source <NUM> queries the BIER controller <NUM> and the management of topology data for this purpose is not critical. In an embodiment, based on the information that the source <NUM> receives from the BIER controller <NUM>, the source forwards one or more packets to a set of replicator nodes <NUM>, <NUM> within the service mesh. The forwarding is performed using unicast.

Operation <NUM>. Each BIER replicator node <NUM>, <NUM> executes a replication of the traffic that it receives from the source <NUM>. The replication is based on the bitmask that is defined by the BIER controller <NUM> based on the receivers <NUM>, <NUM>, <NUM> for a specific multicast group. Each replicator node <NUM>, <NUM> uses in-band policy enforcement to dynamically adjust replication operations. Each replicator node <NUM>, <NUM> may comprise a virtual forwarder engine, virtual switch or other traffic forwarder that allows forwarding traffic based on a vector graph tree or other data structure, and which is programmed to update the BIER controller <NUM> with information about join requests for multicast groups that originate at receivers <NUM>, <NUM>, <NUM> and are forwarded from BIER receiver nodes <NUM>, <NUM>, <NUM> to BIER replicator nodes <NUM>, <NUM>. Examples of virtual forwarders are described at the domain FD. IO on the internet. Vswitch OVS could be used in one embodiment.

Operation <NUM>. Each BIER receiver <NUM>, <NUM>, <NUM> principally executes two functions. First, each BIER receiver <NUM>, <NUM>, <NUM> is responsible for performing joins of its locally connected applications requesting multicast packets. "Joins," in this context, may refer to IGMP join operations. After receiving joins, a BIER receiver <NUM>, <NUM>, <NUM> informs the BIER controller <NUM> of identifies of specific endpoints that are requesting multicast streams for specified multicast groups. Furthermore, when a BIER receiver <NUM>, <NUM>, <NUM> receives replicated messages from a BIER replicator node <NUM>, <NUM>, the BIER receiver <NUM>, <NUM>, <NUM> forwards the replicated messages to those specific receivers <NUM> that the BIER receiver <NUM>, <NUM>, <NUM> manages.

The foregoing steps define one embodiment of multicast-like forwarding in a service mesh using BIER principles. In an embodiment, replication of traffic within a service mesh may be optimized using modifications of iOAM. In an embodiment, nodes processing iOAM packets or messages are programmed to load the iOAM header with per-hop policy definitions. In one embodiment, policies are defined using JavaScript Object Notation (JSON) and JSON elements are carried in iOAM headers. The following operations denoted A, B, C in <FIG> may be used in one embodiment and illustrate packet flow with modified iOAM to provide in-band policy enforcement and definition.

Operation A. iOAM header is encapsulated and defined with policy information for replicator nodes. The source <NUM> adds an iOAM header with policy definitions that are valid for a particular packet or flow. The policy is later used across the replicator nodes <NUM>, <NUM> to define the circumstances under which packets are replicated and forwarded towards receivers <NUM>, or to other replicator nodes that further distribute the packet.

Operation B. Policy definition is transported via iOAM and is executed within the replicator nodes to dynamically adjust the behavior of the replicator nodes. A particular replicator node <NUM>, <NUM> inspects the policy data. In response to the inspection, the replicator node <NUM>, <NUM> selects either an installed policy based on a policy identifier, or dynamically and on-the-fly installs policy specified in the iOAM header to be used for the packet or flow and any consecutive packets or flows. Unlike prior approaches using BIER in other contexts, this operation is unique in providing for dynamically reading, installing and leveraging policies based on details transmitted as part of the actual data stream using iIOAM.

Operation C. Additional policy information is delivered to assure accurate handling of BIER delivered multicast packets at the source. At the BIER receiver node <NUM>, <NUM>, <NUM> that is closest to the edge of the service mesh and thus closest to one or more of the receivers <NUM>, <NUM>, <NUM>, another replicator node (not shown) performs a final replication operation to forward the packet to a specific receiver <NUM>, <NUM>, <NUM>. This receiver is based either within or across different multiple clouds. The other replicator node may be a standalone node or may be incorporated within the BIER receiver node <NUM>, <NUM>, <NUM>.

In one embodiment, the techniques described herein are used in a multi-cloud environment. In such an environment, an implementation cannot rely on feature parity across clouds to deliver multicast packets. Different clouds may not implement all conventional multicast forwarding protocols. In the solution described herein, there is no need for the underlying network to support multicast delivery. Instead, the process described herein can be controlled and enabled by a tenant or administrator of the multi-cloud environment.

In one embodiment, an intelligent controller may automatically determine placement of replicators in cloud networks based on input parameters that may be statically defined or dynamically determined through machine learning. As an example of an automatically machine learned approach, assume that a source <NUM> located within one cloud network is sending unicast packets to replicator nodes <NUM>, <NUM> across the two different clouds, based on load balancing or optimization techniques. The packets are then duplicated closest to the receivers, both within the private cloud but also in environments used across public clouds. This approach reduces the unnecessary overhead of bandwidth utilization and provides intelligent and dynamic traffic distribution across multiple clouds. Consequently, embodiments can provide automatic replicator node placement, with awareness of sources and receivers, to provide optimized delivery within a multi-cloud service mesh.

In embodiments, traffic flows can use iOAM headers to carry policy information on a per-replicator basis to dynamically and on-demand adjust traffic forwarding behavior. In one embodiment, policy data is stored in an SDN controller (not shown in <FIG>, <FIG>) that manages the BIER-based environment and distributed to the BIER controller <NUM> periodically. The iOAM header provides useful structure to carry either a policy identifier of pre-defined policies on replicator nodes or to carry definitions of policy requirements that are dynamically applied at the replicator. Policy data can include QoS parameters, SLA information or per-tenant/per-service details.

In one embodiment, the iOAM header specifies policy that is expressed according to a human-readable, symbolic policy definition language that can be parsed and implemented by devices independent of their implementation, vendor, manufacturer or operation model. The policy definition language may provide constructs that can be used to define policy parameters that are relevant to traffic, while also accepting optional arguments such as tenant/service identifiers or timeframes for off-peak/peak hours. TABLE <NUM> defines an example policy using JSON as a base language:.

Embodiments improve digital data communication services between software elements operating as micro-services and communicating with one another in complex topologies such as service meshes. In particular, embodiments enable a source, such as a micro-service or other program, to transmit a stream of messages relating to an application using multicast-like techniques even though the micro-services are logically defined in a mesh at Layer <NUM> of the OSI reference model rather than Layer <NUM>, Layer <NUM> or Layer <NUM>. Rather than use or require the use of conventional IP multicast protocols, which may not be present in all networks or not implemented across clouds, the new techniques herein use a BIER controller, BIER replicator nodes and BIER receiver nodes to replicate, receive, and forward application or service traffic to receivers that have joined service multicast groups. Furthermore, the BIER receivers receive IGMP join messages from receivers and update replicator nodes, which then update the BIER controller. This improvement permits large, complex meshes of micro-service programs to communicate efficiently without having to use repetitive unicast messages. The result is that fewer messages traverse all links of the network and fewer CPU cycles, less memory and storage are needed for a single source to reach a large number of receivers.

Embodiments also provide for dynamic distribution of replicator policy, using the iOAM header in a new and previously undefined way to carry either a policy identifier or a policy definition in a symbolic language. This approach allows close coupling of policy to traffic and also carries policy identifiers or definitions in a manner that is efficient and does not require defining a new protocol, new field in a protocol, or new message set. Instead, existing implementations of iOAM encapsulation and de-encapsulation, which exist in network nodes for purposes other than policy definition for service mesh multicast traffic, can be reused in a new way to carry policy for this traffic.

According to one embodiment, the techniques described herein are implemented by at least one computing device. The techniques may be implemented in whole or in part using a combination of at least one server computer and/or other computing devices that are coupled using a network, such as a packet data network. The computing devices may be hard-wired to perform the techniques, or may include digital electronic devices such as at least one application-specific integrated circuit (ASIC) or field programmable gate array (FPGA) that is persistently programmed to perform the techniques, or may include at least one general purpose hardware processor programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such computing devices may also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the described techniques. The computing devices may be server computers, workstations, personal computers, portable computer systems, handheld devices, mobile computing devices, wearable devices, body mounted or implantable devices, smartphones, smart appliances, internetworking devices, autonomous or semi-autonomous devices such as robots or unmanned ground or aerial vehicles, any other electronic device that incorporates hard-wired and/or program logic to implement the described techniques, one or more virtual computing machines or instances in a data center, and/or a network of server computers and/or personal computers.

<FIG> is a block diagram that illustrates an example computer system with which an embodiment may be implemented. In the example of <FIG>, a computer system <NUM> and instructions for implementing the disclosed technologies in hardware, software, or a combination of hardware and software, are represented schematically, for example as boxes and circles, at the same level of detail that is commonly used by persons of ordinary skill in the art to which this disclosure pertains for communicating about computer architecture and computer systems implementations.

Computer system <NUM> includes an input/output (I/O) subsystem <NUM> which may include a bus and/or other communication mechanism(s) for communicating information and/or instructions between the components of the computer system <NUM> over electronic signal paths. The I/O subsystem <NUM> may include an I/O controller, a memory controller and at least one I/O port. The electronic signal paths are represented schematically in the drawings, for example as lines, unidirectional arrows, or bidirectional arrows.

At least one hardware processor <NUM> is coupled to I/O subsystem <NUM> for processing information and instructions. Hardware processor <NUM> may include, for example, a general-purpose microprocessor or microcontroller and/or a special-purpose microprocessor such as an embedded system or a graphics processing unit (GPU) or a digital signal processor or ARM processor. Processor <NUM> may comprise an integrated arithmetic logic unit (ALU) or may be coupled to a separate ALU.

Computer system <NUM> includes one or more units of memory <NUM>, such as a main memory, which is coupled to I/O subsystem <NUM> for electronically digitally storing data and instructions to be executed by processor <NUM>. Memory <NUM> may include volatile memory such as various forms of random-access memory (RAM) or other dynamic storage device. Memory <NUM> also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor <NUM>. Such instructions, when stored in non-transitory computer-readable storage media accessible to processor <NUM>, can render computer system <NUM> into a special-purpose machine that is customized to perform the operations specified in the instructions.

Computer system <NUM> further includes non-volatile memory such as read only memory (ROM) <NUM> or other static storage device coupled to I/O subsystem <NUM> for storing information and instructions for processor <NUM>. The ROM <NUM> may include various forms of programmable ROM (PROM) such as erasable PROM (EPROM) or electrically erasable PROM (EEPROM). A unit of persistent storage <NUM> may include various forms of non-volatile RAM (NVRAM), such as FLASH memory, or solid-state storage, magnetic disk or optical disk such as CD-ROM or DVD-ROM, and may be coupled to I/O subsystem <NUM> for storing information and instructions. Storage <NUM> is an example of a non-transitory computer-readable medium that may be used to store instructions and data which when executed by the processor <NUM> cause performing computer-implemented methods to execute the techniques herein.

The instructions in memory <NUM>, ROM <NUM> or storage <NUM> may comprise one or more sets of instructions that are organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps. The instructions may comprise an operating system and/or system software; one or more libraries to support multimedia, programming or other functions; data protocol instructions or stacks to implement TCP/IP, HTTP or other communication protocols; file format processing instructions to parse or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, intemet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. The instructions may implement a web server, web application server or web client. The instructions may be organized as a presentation layer, application layer and data storage layer such as a relational database system using structured query language (SQL) or no SQL, an object store, a graph database, a flat file system or other data storage.

Computer system <NUM> may be coupled via I/O subsystem <NUM> to at least one output device <NUM>. In one embodiment, output device <NUM> is a digital computer display. Examples of a display that may be used in various embodiments include a touch screen display or a light-emitting diode (LED) display or a liquid crystal display (LCD) or an e-paper display. Computer system <NUM> may include other type(s) of output devices <NUM>, alternatively or in addition to a display device. Examples of other output devices <NUM> include printers, ticket printers, plotters, projectors, sound cards or video cards, speakers, buzzers or piezoelectric devices or other audible devices, lamps or LED or LCD indicators, haptic devices, actuators or servos.

At least one input device <NUM> is coupled to I/O subsystem <NUM> for communicating signals, data, command selections or gestures to processor <NUM>. Examples of input devices <NUM> include touch screens, microphones, still and video digital cameras, alphanumeric and other keys, keypads, keyboards, graphics tablets, image scanners, joysticks, clocks, switches, buttons, dials, slides, and/or various types of sensors such as force sensors, motion sensors, heat sensors, accelerometers, gyroscopes, and inertial measurement unit (IMU) sensors and/or various types of transceivers such as wireless, such as cellular or Wi-Fi, radio frequency (RF) or infrared (IR) transceivers and Global Positioning System (GPS) transceivers.

Another type of input device is a control device <NUM>, which may perform cursor control or other automated control functions such as navigation in a graphical interface on a display screen, alternatively or in addition to input functions. Control device <NUM> may be a touchpad, a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor <NUM> and for controlling cursor movement on output device (e.g., display) <NUM>. The input device may have at least two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. Another type of input device is a wired, wireless, or optical control device such as a joystick, wand, console, steering wheel, pedal, gearshift mechanism or other type of control device. An input device <NUM> may include a combination of multiple different input devices, such as a video camera and a depth sensor.

In another embodiment, computer system <NUM> may comprise an intemet of things (IoT) device in which one or more of the output device <NUM>, input device <NUM>, and control device <NUM> are omitted. Or, in such an embodiment, the input device <NUM> may comprise one or more cameras, motion detectors, thermometers, microphones, seismic detectors, other sensors or detectors, measurement devices or encoders and the output device <NUM> may comprise a special-purpose display such as a single-line LED or LCD display, one or more indicators, a display panel, a meter, a valve, a solenoid, an actuator or a servo.

When computer system <NUM> is a mobile computing device, input device <NUM> may comprise a global positioning system (GPS) receiver coupled to a GPS module that is capable of triangulating to a plurality of GPS satellites, determining and generating geo-location or position data such as latitude-longitude values for a geophysical location of the computer system <NUM>. Output device <NUM> may include hardware, software, firmware and interfaces for generating position reporting packets, notifications, pulse or heartbeat signals, or other recurring data transmissions that specify a position of the computer system <NUM>, alone or in combination with other application-specific data, directed toward host <NUM> or server <NUM>.

Computer system <NUM> may implement the techniques described herein using customized hard-wired logic, at least one ASIC, GPU, or FPGA, firmware and/or program instructions or logic which when loaded and used or executed in combination with the computer system causes or programs the computer system to operate as a special-purpose machine. According to one embodiment, the techniques herein are performed by computer system <NUM> in response to processor <NUM> executing at least one sequence of at least one instruction contained in main memory <NUM>. Such instructions may be read into main memory <NUM> from another storage medium, such as storage <NUM>.

Non-volatile media includes, for example, optical or magnetic disks, such as storage <NUM>. Volatile media includes dynamic memory, such as memory <NUM>. Common forms of storage media include, for example, a hard disk, solid state drive, flash drive, magnetic data storage medium, any optical or physical data storage medium, memory chip, or the like.

For example, transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise a bus of I/O subsystem <NUM>. Transmission media can also take the form of acoustic or light waves, such as those generated during radio-wave and infrared data communications.

Various forms of media may be involved in carrying at least one sequence of at least one instruction to processor <NUM> for execution. The remote computer can load the instructions into its dynamic memory and send the instructions over a communication link such as a fiber optic or coaxial cable or telephone line using a modem. A modem or router local to computer system <NUM> can receive the data on the communication link and convert the data to a format that can be read by computer system <NUM>. For instance, a receiver such as a radio frequency antenna or an infrared detector can receive the data carried in a wireless or optical signal and appropriate circuitry can provide the data to I/O subsystem <NUM> such as place the data on a bus. I/O subsystem <NUM> carries the data to memory <NUM>, from which processor <NUM> retrieves and executes the instructions. The instructions received by memory <NUM> may optionally be stored on storage <NUM> either before or after execution by processor <NUM>.

Communication interface <NUM> provides a two-way data communication coupling to network link(s) <NUM> that are directly or indirectly connected to at least one communication networks, such as a network <NUM> or a public or private cloud on the Internet. For example, communication interface <NUM> may be an Ethernet networking interface, integrated-services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of communications line, for example an Ethernet cable or a metal cable of any kind or a fiber-optic line or a telephone line. Network <NUM> broadly represents a local area network (LAN), wide-area network (WAN), campus network, internetwork or any combination thereof. Communication interface <NUM> may comprise a LAN card to provide a data communication connection to a compatible LAN, or a cellular radiotelephone interface that is wired to send or receive cellular data according to cellular radiotelephone wireless networking standards, or a satellite radio interface that is wired to send or receive digital data according to satellite wireless networking standards. In any such implementation, communication interface <NUM> sends and receives electrical, electromagnetic or optical signals over signal paths that carry digital data streams representing various types of information.

Network link <NUM> typically provides electrical, electromagnetic, or optical data communication directly or through at least one network to other data devices, using, for example, satellite, cellular, Wi-Fi, or BLUETOOTH technology. For example, network link <NUM> may provide a connection through a network <NUM> to a host computer <NUM>.

Furthermore, network link <NUM> may provide a connection through network <NUM> or to other computing devices via internetworking devices and/or computers that are operated by an Intemet Service Provider (ISP) <NUM>. ISP <NUM> provides data communication services through a world-wide packet data communication network represented as intemet <NUM>. A server computer <NUM> may be coupled to intemet <NUM>. Server <NUM> broadly represents any computer, data center, virtual machine or virtual computing instance with or without a hypervisor, or computer executing a containerized program system such as DOCKER or KUBERNETES. Server <NUM> may represent an electronic digital service that is implemented using more than one computer or instance and that is accessed and used by transmitting web services requests, uniform resource locator (URL) strings with parameters in HTTP payloads, API calls, app services calls, or other service calls. Computer system <NUM> and server <NUM> may form elements of a distributed computing system that includes other computers, a processing cluster, server farm or other organization of computers that cooperate to perform tasks or execute applications or services. Server <NUM> may comprise one or more sets of instructions that are organized as modules, methods, objects, functions, routines, or calls. The instructions may be organized as one or more computer programs, operating system services, or application programs including mobile apps. The instructions may comprise an operating system and/or system software; one or more libraries to support multimedia, programming or other functions; data protocol instructions or stacks to implement TCP/IP, HTTP or other communication protocols; file format processing instructions to parse or render files coded using HTML, XML, JPEG, MPEG or PNG; user interface instructions to render or interpret commands for a graphical user interface (GUI), command-line interface or text user interface; application software such as an office suite, intemet access applications, design and manufacturing applications, graphics applications, audio applications, software engineering applications, educational applications, games or miscellaneous applications. Server <NUM> may comprise a web application server that hosts a presentation layer, application layer and data storage layer such as a relational database system using structured query language (SQL) or no SQL, an object store, a graph database, a flat file system or other data storage.

Computer system <NUM> can send messages and receive data and instructions, including program code, through the network(s), network link <NUM> and communication interface <NUM>. In the Intemet example, a server <NUM> might transmit a requested code for an application program through Intemet <NUM>, ISP <NUM>, local network <NUM> and communication interface <NUM>. The received code may be executed by processor <NUM> as it is received, and/or stored in storage <NUM>, or other non-volatile storage for later execution.

Claim 1:
A method comprising:
receiving, at a Bit Index Explicit Replication, BIER, replicator node (<NUM>, <NUM>) that is configured to implement a BIER protocol, from a data source (<NUM>) that implements an application layer micro-service, a multicast stream packet identifying a service-level multicast group address of a service-level multicast group;
replicating, via the BIER replicator node, the multicast stream packet according to the BIER protocol;
transmitting, via the BIER replicator node, two or more replicated multicast stream packets to two or more BIER receiver nodes (<NUM>, <NUM>, <NUM>) that are configured to implement BIER;
transmitting, via the two or more BIER receiver nodes, the two or more replicated multicast stream packets to two or more receivers (<NUM>, <NUM>, <NUM>);
receiving an In-situ Operations, Administration, and Maintenance, iOAM, header that comprises at least one of an identifier of a replicator policy or a definition of a replicator policy expressed in a symbolic language, at the BIER replicator node or one or more of a plurality of other BIER replicator nodes; and
at a particular one of the BIER replicator node or the plurality of other BIER replicator nodes, performing at least one of:
based on the identifier of the replicator policy, retrieving a pre-defined packet replication policy that matches the identifier, and executing the pre-defined packet replication policy to dynamically adjust packet processing behavior of the particular one of the BIER replicator nodes; or
parsing the definition of the replicator policy in the symbolic language to yield a new packet replication policy, and executing the new packet replication policy to dynamically adjust packet processing behavior of the particular one of the BIER replicator nodes.