Patent Description:
Current and future generations of edge cloud architectures have capabilities to efficiently and flexibly connect multiple functions and services using pooled resources. For example a cloud services provider may provide pooled memory that may be accessed by multiple functions and/or services through one or more edge systems to efficiently process workloads. While individual client systems have capabilities to effectively monitor operation using performance monitoring tools, such monitoring becomes difficult in an edge cloud architecture where flexible data movement and sharing across multiple edge cloud entities may occur. Document <CIT>discloses a method for monitoring usage metrics including CPU usage and memory usage of a server, wherein a policy agent included in the server receives policies from a policy controller for monitoring the usage metrics according to the policies deployed at the server by the policy agent.

The invention provides subject-matter as defined in the independent claims, preferred embodiments thereof defined in the dependent claims.

Examples of apparatuses, methods, and computer-readable storage mediums are detailed as follows. Features of the present disclosure are recited.

According to one aspect of the present disclosure, an apparatus comprising a monitor circuit to monitor traffic of a plurality of sources through the apparatus and maintain telemetry information regarding the traffic based at least in part on telemetry rules received from the plurality of sources, where the monitor circuit is to determine whether to send a callback message to a selected one of the plurality of sources, in response to telemetry information associated with the selected source and based on a telemetry rule for the selected source, the callback message including the telemetry information associated with the traffic of the selected source through the apparatus; and a storage coupled to the monitor circuit, the storage to store the telemetry information, where the monitor circuit is to access the telemetry information from the storage.

In some examples, the apparatus may further comprise a telemetry rule storage, the telemetry rule storage including a plurality of entries each to store an identifier associated with a source, monitoring rule metadata and tracing event information.

In some examples, the apparatus, wherein the monitor circuit may access a first entry of the telemetry rule storage associated with the selected source to determine whether to send the callback message.

In some examples, the apparatus may further comprise a tracing circuit to trace a first traffic flow from the selected source in response to a tracing request from the monitor circuit.

In some examples, the apparatus, wherein the monitor circuit may send the tracing request based on the tracing event information in the first entry, and further in response to the determination to send the callback message.

In some examples, the apparatus, wherein the tracing circuit may trace a plurality of messages from the selected source to a first destination in response to the tracing request.

In some examples, the apparatus, the tracing circuit may comprise one or more processing circuits to execute one or more bitstreams to trace the first traffic flow.

In some examples, the apparatus, the tracing circuit may comprise one or more processing circuits to execute a correlation bitstream to correlate the first traffic flow with source telemetry information received from the selected source.

In some examples, the apparatus, wherein the monitor circuit may monitor the traffic of the selected source to a first destination and not monitor the traffic of the selected source to a second destination, based on the first entry of the telemetry rule storage associated with the selected source.

In some examples, the apparatus may comprise a switch coupled to the plurality of sources via a plurality of links, where the switch is to receive traffic of a plurality of communication protocols from the selected source via a corresponding one of the plurality of links.

According to one aspect of the present disclosure, a method comprises: receiving, in a switch coupled to a plurality of devices, telemetry rule registration requests from at least some of the plurality of devices; storing a telemetry rule for each of the telemetry rule registration requests in a telemetry rule database; monitoring traffic, through the switch, associated with the at least some of the plurality of devices and maintaining telemetry information regarding the traffic; and in response to the telemetry information associated with a first device of the at least some of the plurality of devices and based on a first telemetry rule for the first device, sending a callback message to the first device, the callback message including at least some of the telemetry information associated with the first device.

In some examples, the method may further comprise in response to the first telemetry rule for the first device, sending a tracing request to a trace circuit of the switch to cause the trace circuit to trace message traffic between the first device and a second device.

In some examples, the method may further comprise sending tracing information associated with the message traffic between the first device and the second device to a telemetry server coupled to the switch.

In some examples, the method may further comprise based on the first telemetry rule, monitoring the traffic through the switch from the first device of a first communication protocol and not monitoring the traffic through the switch from the first device of a second communication protocol.

In some examples, the method may further comprise: routing the traffic of the first communication protocol from the first device to a second device coupled to the switch; and routing the traffic of the second communication protocol from the first device to a third device coupled to the switch.

According to one aspect of the present disclosure, a computer readable medium including instructions is to perform the method of any of the above examples.

In various embodiments, cloud-based edge architectures are provided with monitoring capabilities to flexibly and effectively monitor traffic traveling through such architectures. To this end, various sources that communicate with a cloud-based edge architecture may specify advanced monitoring rules. In turn, monitoring circuitry may perform monitoring of traffic flow at a fine-grained level. In this way, specific traffic patterns and effects on resource usage can be monitored. Although embodiments are not limited in this regard, example cloud-based edge architectures may communicate using interconnects and switches in accordance with a Compute Express Link (CXL) specification such as the CXL <NUM> Specification or any future versions, modifications variations or alternatives to a CXL specification.

Further, while an example embodiment described herein is in connection with CXL-based technology, embodiments may be used in other coherent interconnect technologies such as an IBM XBus protocol, an Nvidia NVLink protocol, an AMD Infinity Fabric protocol, cache coherent interconnect for accelerators (CCIX) protocol or coherent accelerator processor interface (OpenCAPI).

In a CXL implementation, fine-grained monitoring of traffic flows of different communication protocols that are sent along CXL interconnects can occur. For example, there may be separate traffic flows including so-called CXL. cache, CXL. io and CXL. mem communication protocols that can be finely monitored. For example, a given entity may register a monitoring rule to dictate a particular communication protocol to be monitored, as well as identification of source/destination pairs for which monitoring is to be applied. Still further, monitoring can be controlled to enable monitoring for only certain intervals of time, certain address ranges, and so forth.

More generally, embodiments may be used to monitor traffic including communication of data and messages via multiple interconnect protocols, including a CXL protocol as described herein. For example, the interconnect may support various interconnect protocols, including a non-coherent interconnect protocol, a coherent interconnect protocol, and a memory interconnect protocol. Non-limiting examples of supported interconnect protocols may include PCI, PCIe, USB, IDI, IOSF, SMI, SMI3, SATA, CXL. cache, and CXL. mem, and/or the like.

Referring now to <FIG>, shown is a block diagram of a switch in accordance with an embodiment. More specifically, switch <NUM> is a CXL switch to couple to a variety of devices including at least one host device and one or more other devices. Further, while <FIG> shows a high level view of a CXL switch, understand that embodiments are not limited in this regard and similar circuitry may be incorporated in other devices including other switch types, as well as a variety of other devices to leverage embodiments as described herein.

As illustrated, switch <NUM> includes an ingress circuit <NUM> that is configured to receive incoming packets from one or more devices, such as a host system and one or more other devices. In general, ingress circuit <NUM> may perform various processing on the packets. After such processing, the processed packets are provided to a monitor circuit <NUM>. As illustrated, monitor circuit <NUM> may include one or more interfaces <NUM>, a monitor control circuit <NUM>, and a telemetry rule storage <NUM>. Interfaces <NUM> may be exposed to various software stacks running on devices or compute platforms connected to switch <NUM>. Embodiments may enable communication between such external devices and switch <NUM> in order to configure monitoring. In embodiments, only software stacks having the right level of privilege may have access to such monitoring resources, such as may be done via certificate authentication or another authentication method.

In general, monitor circuit <NUM> is configured to process each incoming request against a set of telemetry rules. Based on this analysis, monitor circuit <NUM> may generate callbacks if needed or activate tracing. After processing within monitor circuit <NUM>, packets may be passed to an egress circuit <NUM>, where they may be directed along towards a destination. Note that in some implementations, processing within monitor circuit <NUM> may add too much overhead to each transaction if performed in-line. To this end, ingress circuit <NUM> may copy messages to monitor circuit <NUM> before sending them to egress circuit <NUM>. In this case, monitor circuit <NUM> may include buffering elements to buffer the requests to be processed. It could even consider dropping some of the requests, in an embodiment.

Shown in inset in <FIG> are example entries in telemetry rule storage <NUM>. As illustrated, each entry may include multiple fields, including an ID field <NUM>, an interval field <NUM>, an address range field <NUM>, a metadata field <NUM>, and a tracing event field <NUM>. In an embodiment, each rule thus is defined by: a unique ID to be stored in ID field <NUM>, which is assigned by monitor control circuit <NUM> and returned as part of this call; a list of intervals stored in interval field <NUM> that assert the rule and the metric associated to that rule (e.g., this could [<NUM>,<NUM>] gigabytes per second (GBs) or [<NUM>,N] nanoseconds (ns) of latency); a list of address ranges stored in address field <NUM> to be monitored by that flow (in this case, only requests matching those ranges apply to the intervals); source, destination and protocol to monitor, as stored in field <NUM>, and tracing event information stored in tracing event field <NUM>. In an embodiment, both source and destination may be defined by a platform ID and process address space ID (PASID) or any other field that is part of the request that can be used to identify the source of that request and the target. Tracing event information may be used to identify whether the rule is to start, stop or do nothing with respect to tracing that flow. In an embodiment, interfaces <NUM> also may enable update or deletion of any existing rule within telemetry rule storage <NUM>, e.g., using a corresponding identifier of the rule, which may be a universally unique ID (UUID).

In addition, monitor circuit <NUM> is configured to perform monitoring regarding traffic through switch <NUM> based at least in part on the telemetry rules stored in telemetry rule storage <NUM>. To this end, monitor circuit <NUM> may maintain, e.g., in a monitor storage within monitor circuit <NUM> or coupled thereto telemetry information regarding traffic from multiple sources for flows that fall within the telemetry rules stored in telemetry rule storage <NUM>. As telemetry information may take the form of one or more data sets, these portions of which can be provided to various requesters such as in connection with a callback.

As further illustrated in <FIG>, monitor circuit <NUM> couples to a tracing circuit <NUM>. In embodiments herein, when monitor circuit <NUM> determines that a given packet is subject to a tracing event, e.g., as determined based on one or more rules present in telemetry rule storage <NUM>, monitor circuit <NUM> may send the packet to tracing circuit <NUM> to process. Although embodiments are not limited in this regard, such tracing events may include a tracing start or tracing stop, among others. In addition, based on processing in monitoring control circuit <NUM>, one or more data sets also may be sent to tracing circuit <NUM>.

Tracing circuit <NUM> may be configured to generate traces that can be accessed at any time by the different authenticated software stacks or elements connected to switch <NUM>. Understand that while tracing circuit <NUM> is shown included inside switch <NUM>, in other embodiments the tracing circuit could be another discrete device connected to the switch itself.

As illustrated, tracing circuit <NUM> includes an access interface circuit <NUM> that may direct tracing events and/or data sets to appropriate destinations within tracing circuit <NUM>. To this end, access interface circuit <NUM> may provide an interface that allows access to existing tracing data stored in switch <NUM>. To this end, tracing circuit <NUM> may include one or more storages <NUM> to store various tracing information. Storage <NUM> may, in an environment be separated into multiple independent storages, each associated with a type of tracing data. In the embodiment shown in <FIG>, such storages may include warm storage <NUM>, a cold storage <NUM>, and a hot storage <NUM>.

As examples, access interface circuit <NUM> may provide information as to: type of data to be accessed (e.g., warm, cold and hot); time stamps or range; and delete on access capabilities. In an embodiment, hot tracing may be used to generate more summarized data (hence having less footprint but can be accessed quickly). In turn, warm tracing may have more detail and be accessed slower, while cold tracing is closer to (or is) raw data coming from monitor circuit <NUM>. Note that more than three levels could be present in an embodiment. Access interface circuit <NUM> also may enable registering a set of binaries or bit-streams <NUM> that can run on compute elements or accelerators <NUM> to process the data coming from monitor circuit <NUM>. In addition to such bitstreams or binaries to perform tracing regarding message flows and data set processing, an additional correlation bitstream <NUM> may be provided. In embodiments, correlation bitstream <NUM> may execute on a given computing element/accelerator <NUM> to correlate information received from a given source with the monitoring data information to correlate activity within a given source with traffic flow through switch <NUM>. Understand while shown at this high level in the embodiment of <FIG>, many variations and alternatives are possible.

For example, it is possible to provide alternate storage locations for pre-processed tracing data, such as one or more external storages to store this information. Referring now to <FIG>, shown is a block diagram of a system in accordance with an embodiment. As illustrated in <FIG>, in a system <NUM>, which may be part of an edge appliance, data center architecture or so forth, switch <NUM> is shown coupled to a host <NUM> and multiple devices <NUM>, <NUM>, and <NUM>. As illustrated, device <NUM> may act as a host or another device including a memory <NUM> to store telemetry information. As illustrated, this information may be sent to device <NUM> via a CXL. mem communication protocol. In turn, device <NUM> may act as a host or another device including a media <NUM> to store telemetry information. In different cases, media storage (either memory or storage) may be hosted in switch <NUM>. Understand that one or potentially more media storage-based block type of ranges may be included in these storages. In some cases, tracing may be stored in a circular way for the memory range. If there is any overflow on the allocated space, an interrupt or callback to a management stack can be provided.

With an arrangement as in <FIG>, one or potentially more memory ranges exposed in one or more devices or hosts can store tracing data. In this way, particular hosts or device can request that particular telemetry from switch <NUM> goes into a local address space that they can access.

In the embodiment of <FIG>, one or more hosts or devices may include out of band channels (either specific lanes or authenticated channels) that allow certain or all host and devices to push telemetry information into switch <NUM>. In this way, embodiments may enable correlation schemes between this traffic from the host/device and CXL traffic. To this end, switch <NUM>, via correlation bitstream <NUM>, may perform correlations between activity in host/device and the traffic received in switch <NUM>. As examples, correlation bitstream <NUM> may include various models such principal component analysis (PCA), Markovian, etc..

Referring now to <FIG>, shown is a flow diagram of a method in accordance with an embodiment. More specifically, method <NUM> of <FIG> is a method for configuring a monitor circuit with telemetry rules for a given entity. As an example, this entity may be a device, platform or other hardware coupled to a switch including a monitor circuit in accordance with an embodiment. As such, method <NUM> may be performed by hardware circuitry, firmware, software and/or combinations thereof.

Method <NUM> begins by receiving a monitoring configuration request from an entity in the monitor circuit (block <NUM>). In an embodiment, this request may be received via a CXL link, and more particularly the request may be according to a CXL. io Still with reference to <FIG>, next assuming that the entity is authorized for monitoring, control passes to block <NUM> where a telemetry rule may be inserted. More specifically, this telemetry rule may be inserted into an entry of a telemetry rule table. Such table may be stored in a storage of the monitor circuit or in another circuit coupled to the monitor circuit. This entry may include information in various fields, such as the fields as illustrated in <FIG> above. To enable the requesting entity to thereafter send information to be associated with this telemetry rule, the switch may send an identifier of the telemetry rule to the requesting entity (block <NUM>). At this point, the monitor circuit is appropriately configured to monitor incoming communications from the requesting entity and accordingly, method <NUM> may conclude. And if the entity is not authorized, the request may be dropped at block <NUM>. Understand while shown at this high level in the embodiment of <FIG>, many variations and alternatives are possible.

Referring now to <FIG>, shown is a flow diagram of a method in accordance with another embodiment. More specifically, method <NUM> of <FIG> is a method for operating a monitor circuit to handle incoming transaction requests based on telemetry rules in accordance with an embodiment. As such, method <NUM> may be performed by hardware circuitry, firmware, software and/or combinations thereof.

As shown, method <NUM> begins by receiving a transaction request in a monitor circuit (block <NUM>). In an embodiment, such request may be received from a given device or host coupled to the switch. Understand that this transaction request is from a device that has at least one telemetry rule requested within the monitor circuit. Accordingly, next at diamond <NUM> the monitor circuit may determine whether the transaction request is within an interval to be monitored. This determination may be based upon information in at least one telemetry rule and the request itself. If so, control next passes to diamond <NUM> where the monitor circuit may determine whether the transaction request is within an address range to be monitored. If so, control next passes to diamond <NUM> where the monitor circuit may determine whether the transaction request is for a source/destination combination to be monitored. If the transaction request is not within all of the interval, address range and source/destination combination to be monitored, no further operation occurs.

When it is determined that the transaction request is one for which monitoring is to occur, the monitor circuit may monitor activity with regard to this transaction request and maintain telemetry information (e.g., as to bandwidth, cache latency, etc.). Thus at block <NUM>, the monitor circuit may update monitoring data based on the transaction request. For example, one or more counters may be incremented, timers updated or so forth.

Still with reference to <FIG>, control next passes from block <NUM> to diamond <NUM> to determine whether a callback is in order for this transaction request. As one example, the callback may be indicated if a given monitoring data exceeds (or falls below) a threshold, as specified in a telemetry rule. If so, at block <NUM> a callback may be sent with monitoring data. Accordingly, method <NUM> may conclude. Otherwise if a callback is not in order, next it may be determined whether tracing for this transaction request is to occur (diamond <NUM>). If so, control passes to block <NUM> where a tracing request may be sent to a tracing circuit from the monitor circuit. As such, this transaction request may undergo tracing within the tracing circuit. Understand while shown at this high level in the embodiment of <FIG>, many variations and alternatives are possible. For example, while <FIG> shows an implementation in which a determination as to tracing activity is predicated on the transaction request to request initiation (or termination) of a tracing operation, in other cases this tracing activity may occur in response to information in a telemetry rule. As one example, based on a given telemetry rule, tracing activity may occur in response to a callback.

Referring now to <FIG>, shown is a flow diagram of a method in accordance with yet another embodiment. More specifically, method <NUM> of <FIG> is a method for performing tracing in a tracing circuit in accordance with an embodiment. As such, method <NUM> may be performed by hardware circuitry, firmware, software and/or combinations thereof.

As shown in <FIG>, method <NUM> begins by receiving a tracing request in a tracing circuit from a monitor circuit (block <NUM>). Next, control passes to block <NUM> where a type of tracing may be identified based on the tracing request. In this regard, the monitor circuit may identify a particular type of tracing request, e.g., to be one of a hot, warm or cold tracing operation. Then at block <NUM> the tracing circuit may direct the tracing request to a selected compute element. More specifically, based on the type of tracing request, the tracing circuit may direct the tracing request to a given one of multiple compute elements/accelerators that have been programmed with a corresponding bitstream or binary to handle the particular tracing request. Then at block <NUM> this compute element/accelerator may pre-process the telemetry data. Finally, at block <NUM> the telemetry data may be stored in a selected storage medium. This storage may be included within the tracing circuit itself, within a switch including the tracing circuit, or a device coupled to the switch that has the storage medium. Understand while shown at this high level in the embodiment of <FIG>, many variations and alternatives are possible.

Referring now to <FIG>, shown is a block diagram of a system in accordance with an embodiment. As shown in <FIG>, system <NUM> may be part of a data center architecture that includes a large plurality of servers, storage devices, accelerators, pooled memories among many other such components. In the high level shown in <FIG>, a virtual edge appliance <NUM> is illustrated that includes a switch <NUM> in accordance with an embodiment.

As one example, edge appliance <NUM> may provide for edge cloud base station-centric workloads. Such workloads that may be received according to a workflow <NUM>, which as illustrated is a function-based real-time service edge workflow. As seen, incoming traffic into workflow <NUM> may be received from multiple virtual network functions (VNFs) (namely respective functions VNF1 and VNF3). In turn, the traffic through these VNF's passes through a local breakout and asynchronous services <NUM>-<NUM>. In turn, the resulting workflows pass through independent services <NUM> and <NUM> and the additional VNF (VNF2) back to a user.

Note that workflow <NUM> may be performed within virtual edge appliance <NUM>. As illustrated, incoming traffic may be received via switch <NUM>. This traffic may be monitored and traced using embodiments herein. Based upon the traffic and given workflows, note that the various network functions and services may be performed in one or more of a platform <NUM>, a graphics engine <NUM>, an integrated network interface circuit <NUM>, and a pooled memory <NUM>. As shown, a appliance <NUM>. Components in addition to switch <NUM><NUM>. Platform <NUM> as representative of one or more such platforms within edge appliance <NUM>. In an embodiment, platform <NUM> may include a plurality of processors, each of which may be implemented as multi-core processors, along with a system memory, e.g. formed of double data rate memory,, mother such components. Further coupled to switch <NUM> is in the graphics engine <NUM> which in an embodiment may be implemented as a separate graphics card. Graphics engine <NUM> may include graphics processors or other accelerators to perform specialized functions. As shown, graphics engine <NUM> also may include DDR memory. A intelligence network interface circuit <NUM> also couples to switch <NUM>. In an embodiment, Nick <NUM> may include high-bandwidth memory among other components. Finally as shown in <FIG>, a pooled memory <NUM> also couples to switch <NUM>. In an embodiment, pooled memory <NUM> may be implemented with DDR memory and may store incoming payloads or data units from various VNF support other services, which may be stored in a different queues. In turn, such information may be provided to other VNF store services to further processes such information.

With embodiments herein, edge appliance <NUM> may provide for multi-tenancy and may operate with high load variability. Or, to execute various function-based services. Further will be monitoring and tracing implemented within switch <NUM> as described herein various the quality of service metrics may be maintained for these different services to ensure compliance with SLA and other, e.g., resiliency requirements.

With the arrangement in <FIG>, embodiments may efficiently and flexibly connect multiple VNFs and services using pooled resources through switch <NUM>. In this context, edge appliance <NUM> enables flexible data movement and sharing across multiple edge cloud entities. Edge appliance <NUM> is thus a flexible architecture that can execute a variety of functions and/or services. As an example, appliance <NUM> can push payloads or data units (e.g., packets) from one VNF or service to pooled memory <NUM> to a queue meant to be used for consumption for another specific VNF or service or VNF or service or VNF type (e.g., firewall packets). In addition edge appliance <NUM> may pull payloads or data units from pooled memory <NUM> as requested by a specific VNF or service. For example, a VNF may seek access to its pool of memory or storage to retrieve data that has been pushed for it. Alternatively, the VNF or service can pull from the generic queues types (e.g., pull any payload or data unit meant to be consumed by a firewall VNF).

Edge appliance <NUM> may be implemented as a virtual edge platform, with switch <NUM> that connects accelerators, compute and storage to dynamically create (and scale up and down) platforms depending on the service chain requirements. This topology may include subscriber level agreements (SLA) and resiliency requirements between different parts of the topology. Thereby, a tenant may require specific bandwidth between two points of the topology and specific resiliency (e.g., replicate data to two different pooled memory drawers).

With embodiments that implement monitoring and tracing in switch <NUM> or other hardware, mechanisms are provided to monitor and validate SLAs associated with the functions paid for by a tenant. In this way, a cloud services provider and/or its tenants may use these mechanisms to understand resource utilization for the various functions and the SLA behavior. And users and system owners can understand and identify potential glass-jaws for this programming paradigm.

With an embodiment, monitoring and/or tracing information may be provided to enable an understanding of how I/O pooling or CXL-based types of architectures behave under multi-tenant and function-based architectures. Such information may enable effective monitoring of real-time complex flows (e.g., CXL. io/mem/cache) that may happen across thousands of functions from different service chains sharing the fabric.

Referring now to <FIG>, shown is a block diagram of a portion of a data center architecture in accordance with an embodiment. As shown in <FIG>, system <NUM> may be a collection of components implemented as one or more servers of the of a data center. As illustrated, system <NUM> includes a switch <NUM>, e.g., a CXL switch in accordance with an embodiment. By way of switch <NUM>, which acts as a fabric, various components including one or more central processing units (CPUs) <NUM>, <NUM>, one or more special function units, such as graphics processing units (GPUs) <NUM>, <NUM> and a network interface circuit (NIC) <NUM> may communicate with each other. More specifically, these devices, each of which may be implemented as one or more integrated circuits, provide for execution of functions that communicate with other functions in other devices via one of multiple CXL communication protocols. For example, CPU <NUM> may communicate with NIC <NUM> via a CXL. io communication protocol. In turn, CPUs <NUM>,<NUM> may communicate with GPUs <NUM>,<NUM> via a CXL. mem communication protocol. And, CPUs <NUM>,<NUM> may communicate with each other and from CPU <NUM> to GPU <NUM> via a CXL. cache communication protocol, as examples.

Using switch <NUM> in accordance with an embodiment having a monitor circuit <NUM> and a tracing circuit <NUM>, interfaces may be exposed to the various platforms and components to specify advanced monitoring rules. These rules enable: (<NUM>) generation of automatic callbacks to specific software or hardware instances when determined conditions occur; and/or (<NUM>) activate advanced event tracing for certain flows, e.g., when some of the previous callbacks are activated.

For instance, with reference to <FIG>, assume a service implemented with a chain of functions (S1, S2, S4 and S4) in respective devices (<NUM> (A), <NUM> (B), <NUM> (C), <NUM> (D), and <NUM> (E)) that use the flows discussed above, represented as follows: A to B CXL. io; A to D, C CXL. mem / E to C,D CXL. mem; A to E CXL. cache / E to D CXL. With an embodiment, services S1 and S2 may register the following rules shown in Table <NUM>.

With this arrangement and in accordance with the above examples, one or more devices on which the functions and/or services are executed may send, for receipt in a switch, telemetry rule registration requests. In turn, the switch may store a telemetry rule for each of the telemetry rule registration requests in a telemetry rule database. Then during execution of the system including one or flows for these functions/services, the switch may, via a monitor circuit, monitor traffic through the switch associated with such functions/services that execute on at least some of the devices and maintain telemetry information associated with the traffic.

Then, in response to the telemetry information associated with a first device and based on a first telemetry rule for the first device, the switch, via the monitor circuit, may send a callback message to the first device, the callback message including at least some of the telemetry information associated with the first device.

Also in certain cases, such as illustrated in Table <NUM>, in response to the first telemetry rule for the first device, the monitor circuit of the switch may send a tracing request to a trace circuit of the switch to cause the trace circuit to trace message traffic between the first device and a second device. Of course other examples are possible. For example, as further shown in Table <NUM>, it is possible based on comparison of certain monitored information (e.g., cache latency metrics, memory bandwidth metrics or so forth) to one or more thresholds, to implement a callback to send at least some of the monitoring data to one or more services/functions and not implement tracing. And it is equally possible for tracing to be initiated for one or more services/functions to trace messages or other communications, without occurrence of a callback.

Thus embodiments provide very flexible and advanced mechanisms to track SLA, understand how complex flows behave, monitor in a scalable way, and allow identification of glass-jaws, potential bugs or even potential attacks without interfering with service performance. As a result, different levels of smart tracing schemes may be applied depending on requirements of a software stack. And embodiments thus enable, in function-based and edge platforms, fine-grained monitoring capabilities, particularly in cloud-based architectures.

Referring now to <FIG>, shown is a block diagram of a system in accordance with another embodiment of the present invention. As shown in <FIG>, a system <NUM> may be any type of computing device, and in one embodiment may be a server system such as an edge platform. In the embodiment of <FIG>, system <NUM> includes multiple CPUs 810a,b that in turn couple to respective system memories 820a,b which in embodiments may be implemented as double data rate (DDR) memory. Note that CPUs <NUM> may couple together via an interconnect system <NUM> such as an Intel® Ultra Path Interconnect or other processor interconnect technology.

To enable coherent accelerator devices and/or smart adapter devices to couple to CPUs <NUM> by way of potentially multiple communication protocols, a plurality of interconnects 830a1-b2 may be present. In an embodiment, each interconnect <NUM> may be a given instance of a CXL.

In the embodiment shown, respective CPUs <NUM> couple to corresponding field programmable gate arrays (FPGAs)/accelerator devices 850a,b (which may include graphics processing units (GPUs), in one embodiment. In addition CPUs <NUM> also couple to smart NIC devices 860a,b. In turn, smart NIC devices 860a,b couple to switches 880a,b (e.g., CXL switches in accordance with an embodiment) that in turn couple to a pooled memory 890a,b such as a persistent memory. With an arrangement as in <FIG>, CPUs <NUM> may direct certain workloads (e.g., graphics workloads) to devices <NUM> to perform processing on incoming information, which stores processed information in pooled memory <NUM>. In turn, CPUs <NUM> or other entities may access and further process this information from pooled memory <NUM>. And as such flows proceed through switches <NUM>, the fine-grained monitoring and tracing may occur, as described herein.

Turning next to <FIG>, an embodiment of a SoC design in accordance with an embodiment is depicted. As a specific illustrative example, SoC <NUM> may be configured for insertion in any type of computing device, ranging from portable device to server system. Here, SoC <NUM> includes <NUM> cores <NUM> and <NUM>. Cores <NUM> and <NUM> may conform to an Instruction Set Architecture, such as an Intel® Architecture Core™-based processor, an Advanced Micro Devices, Inc. (AMD) processor, a MIPS-based processor, an ARM-based processor design, or a customer thereof, as well as their licensees or adopters. Cores <NUM> and <NUM> are coupled to cache controller <NUM> that is associated with bus interface unit <NUM> and L2 cache <NUM> to communicate with other parts of system <NUM> via an interconnect <NUM>. As seen, bus interface unit <NUM> includes a DMA circuit <NUM> configured to send write requests.

Interconnect <NUM> provides communication channels to the other components, such as a Subscriber Identity Module (SIM) <NUM> to interface with a SIM card, a boot ROM <NUM> to hold boot code for execution by cores <NUM> and <NUM> to initialize and boot SoC <NUM>, a SDRAM controller <NUM> to interface with external memory (e.g., DRAM <NUM>), a flash controller <NUM> to interface with non-volatile memory (e.g., flash <NUM>), a peripheral controller <NUM> (e.g., an eSPI interface) to interface with peripherals, video codec <NUM> and video interface <NUM> to display and receive input (e.g., touch enabled input), GPU <NUM> to perform graphics related computations, etc. In addition, the system illustrates peripherals for communication, such as a Bluetooth module <NUM>, <NUM> modem <NUM>, GPS <NUM>, and WiFi <NUM>. Also included in the system is a power controller <NUM>. Further illustrated in <FIG>, system <NUM> may additionally include interfaces including a MIPI interface <NUM>, e.g., to a display and/or an HDMI interface <NUM> also which may couple to the same or a different display.

Referring now to <FIG>, shown is a block diagram of a system in accordance with another embodiment of the present invention such as an edge platform. As shown in <FIG>, multiprocessor system <NUM> includes a first processor <NUM> and a second processor <NUM> coupled via a point-to-point interconnect <NUM>. As shown in <FIG>, each of processors <NUM> and <NUM> may be many core processors including representative first and second processor cores (i.e., processor cores 1074a and 1074b and processor cores 1084a and 1084b).

In the embodiment of <FIG>, processors <NUM> and <NUM> further include point-to point interconnects <NUM> and <NUM>, which couple via interconnects <NUM> and <NUM> (which may be CXL buses) to switches <NUM> and <NUM>, which may perform fine-grained monitoring and tracing as described herein. In turn, switches <NUM>, <NUM> couple to pooled memories <NUM> and <NUM>. In this way, switches <NUM>, <NUM> may, based on telemetry rules provided by, e.g., processors <NUM> and <NUM>, perform monitoring and tracing of traffic in a fine-grained manner, as described herein.

Still referring to <FIG>, first processor <NUM> further includes a memory controller hub (MCH) <NUM> and point-to-point (P-P) interfaces <NUM> and <NUM>. Similarly, second processor <NUM> includes a MCH <NUM> and P-P interfaces <NUM> and <NUM>. As shown in <FIG>, MCH's <NUM> and <NUM> couple the processors to respective memories, namely a memory <NUM> and a memory <NUM>, which may be portions of system memory (e.g., DRAM) locally attached to the respective processors. First processor <NUM> and second processor <NUM> may be coupled to a chipset <NUM> via P-P interconnects <NUM> and <NUM>, respectively. As shown in <FIG>, chipset <NUM> includes P-P interfaces <NUM> and <NUM>.

Furthermore, chipset <NUM> includes an interface <NUM> to couple chipset <NUM> with a high performance graphics engine <NUM>, by a P-P interconnect <NUM>. As shown in <FIG>, various input/output (I/O) devices <NUM> may be coupled to first bus <NUM>, along with a bus bridge <NUM> which couples first bus <NUM> to a second bus <NUM>. Various devices may be coupled to second bus <NUM> including, for example, a keyboard/mouse <NUM>, communication devices <NUM> and a data storage unit <NUM> such as a disk drive or other mass storage device which may include code <NUM>, in one embodiment. Further, an audio I/O <NUM> may be coupled to second bus <NUM>.

Embodiments as described herein can be used in a wide variety of network architectures. To this end, many different types of computing platforms in a networked architecture that couples between a given edge device and a datacenter can perform the fine-grained monitoring and tracing described herein. Referring now to <FIG>, shown is a block diagram of a network architecture in accordance with another embodiment of the present invention. As shown in <FIG>, network architecture <NUM> includes various computing platforms that may be located in a very wide area, and which have different latencies in communicating with different devices.

In the high level view of <FIG>, network architecture <NUM> includes a representative device <NUM>, such as a smartphone. This device may communicate via different radio access networks (RANs), including a RAN <NUM> and a RAN <NUM>. RAN <NUM> in turn may couple to a platform <NUM>, which may be an edge platform such as a fog/far/near edge platform, and which may leverage embodiments herein. Other requests may be handled by a far edge platform <NUM> coupled to RAN <NUM>, which also may leverage embodiments.

As further illustrated in <FIG>, another near edge platform <NUM> may couple to RANs <NUM>, <NUM>. Note that this near edge platform may be located closer to a data center <NUM>, which may have a large amount of computing resources. By pushing messages to these more remote platforms, greater latency is incurred in handling requests on behalf of edge device <NUM>. Understand that all platforms shown in <FIG> may incorporate embodiments as described herein to perform fine-grained monitoring and tracing of disparate flows.

Note that the terms "circuit" and "circuitry" are used interchangeably herein. As used herein, these terms and the term "logic" are used to refer to alone or in any combination, analog circuitry, digital circuitry, hard wired circuitry, programmable circuitry, processor circuitry, microcontroller circuitry, hardware logic circuitry, state machine circuitry and/or any other type of physical hardware component. Embodiments may be used in many different types of systems. For example, in one embodiment a communication device can be arranged to perform the various methods and techniques described herein. Of course, the scope of the present invention is not limited to a communication device, and instead other embodiments can be directed to other types of apparatus for processing instructions, or one or more machine readable media including instructions that in response to being executed on a computing device, cause the device to carry out one or more of the methods and techniques described herein.

Claim 1:
An apparatus (<NUM>, <NUM>) comprising:
a monitor circuit (<NUM>) to monitor traffic of a plurality of sources through the apparatus and maintain telemetry information regarding the traffic based at least in part on telemetry rules received from the plurality of sources, wherein the monitor circuit is to determine whether to send a callback message to a selected one of the plurality of sources in response to telemetry information associated with the selected source and based on a telemetry rule for the selected source, the callback message including the telemetry information associated with the traffic of the selected source through the apparatus; and
a storage (<NUM>) coupled to the monitor circuit, the storage to store the telemetry information, wherein the monitor circuit is to access the telemetry information from the storage.