Patent Description:
For example, consider an application executing on a single node, such as a server. A primary means to analyze and debug performance for such an application would be to use function-based profiling of the application and examine the "hotspots" or "hot spots," which correspond to region(s) of a computer program where a high proportion of executed instructions occur or where more time (relatively) is spent during the program's execution. For example, function-based profiling tools such as the Intel® VTune Profiler may be used to locate application "hotspots" and observer various related metrics, such as shown in the screenshot in <FIG>. As an illustration, it could be the case that there is a lock-related function that is consuming <NUM>% of CPU time without doing any useful work, and this can easily be identified from the function profile.

Performing this type of analysis becomes significantly more complicated for distributed applications, especially when using a mixture of heterogeneous machines (e.g., servers with different capabilities) interconnected via computer networks with different latencies and capacities and performing different (program) tasks. Worse yet, in today's distributed processing environments a given server may be used for performing concurrent tasks associated with multiple unrelated programs. For example, in multitenant deployments a given physical server's processing resources may be virtualized and leased to multiple separate users, each using their share of the processing resources to execute their own programs.

Prior art document <CIT> discloses that an application testing and analysis may include performing perturbations to affect an environment associated with the application executing on a user device without affecting other applications executing on the user device. The execution of the application may be traced while the perturbations are being performed to determine an amount of resources of the user device consumed by the application and to determine whether a performance of the application was degraded. <CIT> discloses a method for correlating data elements among telemetry datasets, involves processing joined dataset with set of adaptive functions to correlate data entries of joined dataset to data selection, and evaluating associated correlation dataset.

An object of the invention is a method as claimed in claim <NUM> for obtaining telemetry data and tracing data for one or more applications implemented via execution of a plurality of distributed processes on a plurality of compute nodes. Another object of the invention is a compute platform as claimed in claim <NUM>. Preferred embodiments are covered by the appended dependent claims.

The foregoing aspects and many of the attendant advantages of this invention will become more readily appreciated as the same becomes better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified:.

Embodiments of methods and apparatus for scale out hardware-assisted tracing schemes for distributed and scale-out applications are described herein. In the following description, numerous specific details are set forth to provide a thorough understanding of embodiments of the invention.

Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more examples as long as said combinations are encompassed within the scope of the invention as defined by the appended claims.

For clarity, individual components in the Figures herein may also be referred to by their labels in the Figures, rather than by a particular reference number. Additionally, reference numbers referring to a particular type of component (as opposed to a particular component) may be shown with a reference number followed by "(typ)" meaning "typical. " It will be understood that the configuration of these components will be typical of similar components that may exist but are not shown in the drawing Figures for simplicity and clarity or otherwise similar components that are not labeled with separate reference numbers. Conversely, "(typ)" is not to be construed as meaning the component, element, etc. is typically used for its disclosed function, implement, purpose, etc..

How does one employ a similar methodology used for single nodes in scale out architectures, say among N nodes? When the work is distributed amongst N nodes, there should be a holistic means to look at how the application is executing across the N nodes as a whole, and where the application is collectively spending its time in execution. In order to get this, several challenges are encountered, such as:.

A high-level view of an architecture <NUM> in which aspects of the telemetry scheme are implemented for end-to-end hardware tracing support for edge architectures is shown in <FIG>. The top-level components in architecture <NUM> include a data center edge <NUM>, a street cabinet <NUM>, an edge client <NUM>, a tracing service <NUM>, and a switch <NUM>. Data center edge includes multiple servers, such as blade servers, server modules, 1U, 2U servers, etc., and associated devices (e.g., network interfaces and network interface controllers (NICs), storage devices, accelerators, storage class memory (SMC) etc.) installed in one or more cabinets and/or racks. The platforms and devices (and/or their associated resources) may be composed into compute nodes <NUM> configured to support scale-out tracing, as depicted by a platform <NUM>, devices <NUM> and scale-out tracing logic 216a.

Street cabinet <NUM> represents an enclosure (such as a steel box), outbuilding or similar structure housing computing equipment that is located remotely from a data center (e.g., external from data center edge <NUM>). Non-limiting examples of steel cabinet <NUM> uses include telecommunications equipment (such as located in a structure at the base of a cellular tower), city infrastructure equipment, public or private utility company equipment, and computing equipment that may be used for distributed processing used by companies or individuals. Street cabinet <NUM> includes computing resources comprising one or more compute nodes <NUM> configured to support scale-out tracing, as depicted by platform <NUM>, devices <NUM> multiple servers, each comprising a platform <NUM> and includes devices <NUM> and scale-out tracing logic 216b.

Client <NUM>, also labeled and referred to as edge client 'A', represents a client machine (e.g., desktop, laptop, notebook, workstation) that accesses one or more services provided by data center edge <NUM> using a client service application <NUM> and applicable protocols over a network (see <FIG> below), such as HTTP/HTTPS, REST, JSON, etc. Under a common use case, client services provided by data center edge <NUM> are implemented as distributed services hosted on multiple compute nodes or platforms, which may include one or more compute nodes <NUM> in data center edge <NUM> plus (optionally) one or more compute nodes <NUM> in street cabinet <NUM>. In some instances, a client machine itself (e.g., edge client <NUM>) may be involved in performing a distributed service, process, etc. Accordingly, edge client <NUM> also includes scale-out tracing logic 216c.

In accordance with an aspect of some implementations, telemetry data relating to traffic (data transfers) between applications running on compute nodes in one or more street cabinets and a data center (e.g., between applications running on compute node <NUM> in street cabinet <NUM> and data center edge <NUM>) may be traced. Accordingly, in some implementations an intermediate network device (i.e., a device along a network datapath) such as switch <NUM> is configured with scale-out tracing logic 216d.

Generally, an implementation will include tracing service <NUM> with some form of aggregator <NUM> and Network Timing Protocol (NTP) logic <NUM>. In some implementations tracing service <NUM> may be hosted on a server of platform host that is external to a data center (such as illustrated in <FIG>), while in other implementations tracing service is hosted by compute resources in a data center.

Current telemetry measures within a platform are usually attached to a PASID (process application space ID), which may be associated with one or process instances. To expand the PASID concept to scale-out telemetries, a global group ID (GGID) is used to map a set of applications or processes together. Architecture <NUM> proposes to use the GGID plus PASID when generating telemetry and tracing within each of the different elements in a data center (platform, devices, switches etc.). During the lifetime of a set of processes logically belonging to the same workload, in one embodiment the new telemetry logic will:.

An exemplary use of a GGID and PASIDs are shown in <FIG>, which depicts a group of services <NUM> that are assigned a GGID = <NUM>. The group of services includes a Service 'A' with a PASID = <NUM> that is running on one or more compute nodes <NUM> in street cabinet <NUM>, a Service 'B' with a PASID = <NUM> running on one or more compute nodes <NUM> in data center edge <NUM>, and a Service 'C' with a PASID = <NUM> running on one or more compute nodes <NUM> in street cabinet <NUM>. The group of service further includes client service <NUM>, which has a PASID = <NUM>.

<FIG> shows a distributed compute node topology <NUM> illustrating additional aspects to those shown in architecture <NUM> of <FIG>. Compute node topology includes one or more data centers 302a. 302n, interconnected by a private network <NUM>. Each data center 302a. 302n includes a plurality of compute nodes <NUM> hosted on individual platforms and/or virtualized compute nodes composed of virtualized compute, memory, and storage resources. For example, a data center may employ conventional rack architectures comprising servers or the like with integrated resources (e.g., a blade server with CPU, memory, storage, NIC, etc.), or employ disaggregated architectures such as Intel® Rack Scale Design under which compute, storage, accelerator, and other resources are pooled and reside in separate "drawers" or chassis that are interconnected via a high-speed fabric. As further illustrated, an instance of scale-out tracing logic <NUM> is implemented on each compute node <NUM>, wherein the illustrated compute node represent compute nodes in a data center that are used for particular distributed processing tasks for which the telemetry scheme disclosed herein may be implemented. Compute nodes <NUM> are interconnected via a network <NUM> including one or more switches <NUM>. In some embodiments, one or more switches within a data center, such as switch <NUM>, are configured with an instance of scale-out tracing logic <NUM>.

Some distributed compute node topologies may be implemented entirely in data centers, while other distributed compute node topologies may employ compute nodes that are not implemented in data centers, such as compute nodes in street cabinets and/or compute nodes operated by individual persons and public and private institutions (e.g., universities) and companies. For example, distributed compute node topology <NUM> includes street cabinets 304a (also referred to as Street Cabinet 'A') and a street cabinet 304b (also referred to as Street Cabinet 'B'). Street cabinet 304a includes a single compute node comprising a standalone server <NUM> or similar compute platform, while street cabinet 304b includes a cluster of servers <NUM>. An instance of scale-out tracing logic <NUM> is implemented on each of server <NUM> and servers <NUM>.

Distributed compute node topology <NUM> also includes one or more clients <NUM> running (a) client application(s) on various types of platforms, such as a workstation <NUM> (depicted), laptop, notebook, server, mobile device, etc. In some instances, client <NUM> will be operated by a user <NUM>. For example, a user may use a client application to monitor telemetry data aggregated by a tracing service <NUM> or aggregated through other means. Clients may also be used as compute resources for performing distributed tasks. In this case, a client may be operated programmatically or remotely, or may be operated by a user.

Distributed compute node topology <NUM> further depicts some exemplary network infrastructure and paths that are used to interconnect the various compute nodes. In the illustrated embodiment, client <NUM> is connected to a Web server <NUM> at the edge of data center 302a via the Internet <NUM>. In some embodiments, communication between client <NUM> and Web server <NUM> uses HTTPS and/or a Virtual Private Network (VPN) connection. In some embodiments, an instance of scale-out tracing logic <NUM> is implemented on Web server <NUM>.

Street cabinets 304a and 304b are connected to a hybrid server <NUM> at the edge of data center 302a via a private network <NUM>. For example, a mobile service operator may lease network communication infrastructure (or deploy their own infrastructure) to connect its cellular towers to one or more datacenters. Private network <NUM> includes a switch <NUM> that is configured with an instance of scale-out tracing logic <NUM>. As further shown, tracing service <NUM> may be hosted on one or more compute platforms coupled to private network <NUM> or may be hosted using compute resources in data center 302a. In some embodiments, an instance of scale-out tracing logic <NUM> is implemented on hybrid server <NUM>.

In some instances, distributed processing will be performed by compute nodes in two or more data centers that are interconnected via applicable data center edge components configured to communicate via one or more networks, such as private network <NUM> in <FIG>. In some implementations, one or more switches <NUM> in private network <NUM> are configured with an instance of scale-out tracing logic <NUM>.

It is noted that the use of instances of scale-out tracing logic <NUM> in <FIG> is not to imply that each instance of scale-out tracing logic implemented by the various components and apparatus (e.g., servers, switches, compute nodes, etc.) is the same, but is rather used for convenience and simplicity. In practice, the configuration and operation of a given instance of scale-out tracing logic <NUM> will depend on its particular use and implementation context.

<FIG> shows an architecture <NUM> illustrating further details of components used to implement the functionality associated with scale-out tracing logic and associated components. Architecture <NUM> includes a hardware-based telemetry block <NUM> comprising configuration interfaces <NUM>, advanced scale-out telemetry logic <NUM> and NTP logic <NUM>. Advanced scale-out telemetry <NUM> includes monitoring logic <NUM>, performance monitoring (PMON) data logic <NUM>, and PASID to GGID mapping logic <NUM>. Advanced scale-out telemetry <NUM> may also be configured to provide collected and processed telemetry data to scale-out tracing server <NUM> for embodiments employing a scale-out tracing server.

In the illustrated embodiment of <FIG>, telemetry block <NUM> is implemented on each of one or more processors <NUM> installed in a platform, further details of which are shown in <FIG>. In one embodiment the processor has a System on Chip architecture, as depicted by processor/SoC <NUM>, which includes iM processor cores <NUM>, such as <NUM>, <NUM>, <NUM>, <NUM> or more cores. In addition to telemetry block <NUM>, processor/SoC <NUM> also includes a power block <NUM>, a performance monitoring unit (PMU) <NUM>, an integrated memory controller <NUM>, and other blocks for which performance monitoring (PMON) data may be provided, as collectively depicted by an "other" block <NUM>.

Modern processors include various PMON blocks that collect performance monitoring data such as telemetry data and/or tracing data from associated components on the processor. In <FIG> these include a PMON block <NUM> for each processor core <NUM>, and PMON blocks <NUM>, <NUM>, and <NUM>. A processor/SoC will generally include further components, such as various interconnects, input-output (IO) interfaces, an IO Memory Management Unit (IOMMU), etc., that are not shown for simplicity.

During operation, advanced scale-out telemetry <NUM> receives PMON data <NUM> and other data from PMON blocks <NUM>, <NUM>, <NUM>, <NUM>, platform resources <NUM>, and a NIC <NUM>. The platform resources may include double data-rate (DDR) random access memory (RAM) (depicted as DDR memory <NUM>) and HBM <NUM>. DDR memory includes but is not limited to DDR4 and DDR5 memory. In addition, other types of memory may be used, including various types of volatile (e.g., Dynamic RAM (DRAM and Synchronous Dynamic RAM (SDRAM), and non-volatile memory. Memory device form factors include but are not limited to Dual Inline Memory Modules (DIMMs) and Non-Volatile DIMMs (NVDIMMs).

Non-volatile memory is a storage medium that does not require power to maintain the state of data stored by the medium. Non-limiting examples of nonvolatile memory may include any or a combination of: solid state memory (such as planar or 3D NAND flash memory or NOR flash memory), 3D crosspoint memory, storage devices that use chalcogenide phase change material (e.g., chalcogenide glass), byte addressable nonvolatile memory devices, ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS) memory, polymer memory (e.g., ferroelectric polymer memory), ferroelectric transistor random access memory (Fe-TRAM) ovonic memory, nanowire memory, electrically erasable programmable read-only memory (EEPROM), other various types of non-volatile random access memories (RAMs), and magnetic storage memory. In some embodiments, 3D crosspoint memory may comprise a transistor-less stackable cross point architecture in which memory cells sit at the intersection of words lines and bit lines and are individually addressable and in which bit storage is based on a change in bulk resistance. In particular embodiments, a memory module with non-volatile memory may comply with one or more standards promulgated by the Joint Electron Device Engineering Council (JEDEC), such as JESD218, JESD219, JESD220-<NUM>, JESD223B, JESD223-<NUM>, or other suitable standard (the JEDEC standards cited herein are available at www.

Also depicted are processes <NUM> and <NUM> (also labeled and referred to as process A and process B, respectively) which are executing on one or more processor cores <NUM> (e.g., on Core <NUM> and Core <NUM> in <FIG>. The bandwidth utilization of process A and process B are depicted for each of NIC <NUM>, DDR <NUM>, and HBM <NUM>, where the crosshatch patterns are used to convey what bandwidth is associated with which process. For example, process A consumes/employs <NUM> Megabits per second (Mbs) of bandwidth for NIC <NUM>, <NUM> MBs of bandwidth for DDR memory <NUM>, and <NUM> MBs of bandwidth for HBM <NUM>. Meanwhile, process B consumes/employs <NUM> Mbs of bandwidth for NIC <NUM>, <NUM> MBs of bandwidth for DDR memory <NUM>, and <NUM> MBs of bandwidth for HBM <NUM>. The illustrated process bandwidths are merely exemplary, and it is envisioned that upwards of hundreds of processes may be implemented concurrently on a given platform.

A set of interfaces including configuration interfaces <NUM> supports confirmation of and access to the scale-out telemetry scheme and associated data. One interface supports mapping of one or more processes identified by one or multiple PASIDS to a global group of telemetry. In one embodiment, the following parameters are provided:.

Another interface (exposed in an out-of-band fashion with optional authentication in one embodiment) is used to access telemetry for a particular Global Group ID. In one embodiment this interface includes:.

In one embodiment, advanced scale-out telemetry logic <NUM> is responsible for managing and collecting the telemetry data for a particular GGID. Monitoring logic <NUM> is responsible for storing telemetry data coming from the different elements of the platform (such as processor cores) and mapping/associating the telemetry data by its PASID and GGID. In one embodiment, existing interfaces (e.g. collectd) may be used to define what performance counters or telemetry is to be collected for a PASID. Monitoring logic <NUM> uses NTP logic <NUM> to get the current time stamp for the GG before storing the data. Depending on the configuration for a given GGID, monitoring logic <NUM> will forward the telemetry data to PMON Data logic <NUM> or send the telemetry data to scale-out tracing server <NUM>.

In one embodiment, PMON data logic <NUM> is responsible for storing the data collected by monitoring logic <NUM>. As shown, PMON data logic <NUM> indexes this data by GGID and PASID. PASID to GG mapping logic <NUM> is the logic used to map PASIDs to GGIDs. For example, multiple PASIDs may be mapped to a given GGID. This is configured by the first of interface discussed above.

NTP logic <NUM> is responsible for coordinating timing events between the distributed processing elements (e.g., data center compute nodes) where a particular GG has processes running. In one embodiment, NTP logic <NUM> includes a mapping of a list of elements where the GG has a PASID or may be used by processes belonging to the GG (e.g., Service A uses switch X to communicate with Service B). NTP logic <NUM> further may include a mapping of accuracy for each of the different GGs that are registered in the platform. Depending on this accuracy, NTP logic <NUM> may also be responsible for keeping synchronization of the time between different processing elements. For example, in one embodiment NTP logic <NUM> employs the Network Time Protocol (hence the name NTP logic). Other network time or clock synchronization schemes, both existing and future schemes, may also be used.

Scale-out tracing server <NUM> is configured to interface with advanced scale-out telemetry logic <NUM> running on various distributed processing elements and switches and store the various telemetry data that it receives from the distributed processing elements and switches. In one embodiment, scale-out tracing server <NUM> includes:.

In one embodiment, authentication and security schemes are implemented to set up secure communication links or channels between telemetry data providers (e.g., advanced scale-out telemetry logic <NUM> running on a compute node) and the input interfaces of scale-out tracing server <NUM>, and between clients and output interfaces of the scale-out tracing server.

<FIG> shows a flowchart <NUM> illustrating operations associated with obtaining telemetry and tracing data for applications implemented using distributed processes. The operations in blocks <NUM> and <NUM> are performed to configure the distributed processing environment and include distributing and/or deploying application processes to processing elements on a plurality of compute nodes interconnected by one or more networks and/or interconnected by socket-to-socket links. Non-limiting examples of distributed processing environments including compute nodes interconnected via one or more networks are shown in <FIG> and <FIG> and discussed above. Generally, as used herein a compute node is a compute platform, such as a server that includes a processing element on which software processes (aka threads) are executed. Processing elements include processors, such as but not limited to processors manufactured by Intel® Corporation, AMD® Corporation, and various ARM-based processors having ARM® cores and/or architectures. Generally, the software code associated with a process may be loaded into memory from a storage device on the platform or may be loaded over a network from a network storage device or the like. Some compute nodes may be implemented on multi-socketed platforms with two or more processors interconnected via high-speed socket-to-socket interconnect, wherein each processor operates as a compute node.

In block <NUM>, a global time synchronization mechanism is implemented. In one embodiment, the Network Time Protocol is implemented, as discussed above. NTP is a well-known standardized protocol for clock synchronization between computer systems interconnected via packet-switched networks. The NTP standards are available at www. In other embodiments, other time synchronization mechanisms and protocols may be used.

As depicted by start and end loop block <NUM> and <NUM>, the operations in blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM> and optional block516, <NUM> are performed for each of one or more applications implemented via execution of distributed processes for which telemetry and/or tracing data is to be obtained. In a block <NUM>, the processes associated with the application are identified, as well as processing elements on the compute nodes that are used to execute instances of the processes. During the distributed processing operations, instances of processes will be launched. As shown in block <NUM>, each instance of a process will be associated with a PASID.

Each application has an associated GGID for which the distributed processes used to implement the application are associated, as shown in block <NUM>. The associated between GGIDs and PASIDs one individual compute node is implemented by PASID to GGID mapping logic <NUM> in one embodiment.

In block <NUM> telemetry and/or tracing data is obtained from the processing elements on which the process instances are executed. For example, as discussed above with reference to <FIG>, PMON blocks <NUM> are used to collect various performance and statistical data relating to processes executing on processor cores <NUM>. PMON blocks <NUM> also provide interfaces to enable data they collect to be retrieved by monitoring logic <NUM>.

In an optional block <NUM>, telemetry and/or tracing data is obtained from one or more other elements on the compute nodes, such as from a NIC, management component(s) on the platform, such as a baseboard management controller, or any other platform element configured to generate telemetry and/or tracing data that is to be included in the telemetry/tracing data for the application.

In another optional block <NUM>, telemetry and/or tracing data is obtained from one or more network switches. As discussed above, network switches may be configured to generated telemetry data that may be forwarded to a tracing server or one of the compute nodes.

In a block <NUM> timestamps are associated with the telemetry and/or tracing using the global time synchronization mechanism. For example, for compute nodes this may be implemented by NTP logic <NUM>. NTP logic may also be implemented for switches or, optionally, the telemetry/tracing data obtained from a switch may include an NTP timestamp associated with packets used for tracing or providing telemetry data.

As shown in block <NUM>, selected telemetry and/or tracing data for one or more application is accessed. For instance, such data may be accessed by a client accessing a tracing service provided by a tracing server or provided by one or the compute nodes that is implemented for aggregating telemetry and tracing data.

<FIG> shows a flowchart <NUM> illustrating operations to associate processing elements and compute nodes with processes executed on the processing elements and compute nodes. In a block <NUM>, processing elements on the compute nodes are identified by type and/or processing capabilities. For example, processors generally have model or product numbers that may be used to identify a processor and its capabilities. In a block <NUM>, processing element (or processor) identifiers (ProcIDs) and associated with the various processing elements used for executing the processes associated with the one or more distributed applications. Optionally or in addition to, compute node identifiers (NodeIDs) are associated with the compute nodes on which the processing elements reside. Under one embodiment, in cases under which a compute node has a single processing element, the NodeID may be substituted for a ProcID.

In a block <NUM>, the ProcID and/or the NodeID for a given processing element or compute node is associated with telemetry and/or tracing data obtained from the processing element or compute node. In a block <NUM>, the telemetry and tracing data for an application is evaluated in consideration of the type of processing element and/or the processing elements' capabilities.

<FIG> shows a flowchart <NUM> illustrating operations performed to implement aspects of telemetry and tracing data storage, forwarding, aggregation, and presentation, according to one embodiment. In a block <NUM>, telemetry and/or tracing data is stored on the compute nodes indexed by PASIDs and GGIDs, such as described above and illustrated in <FIG>. In a block <NUM>, telemetry and/or tracing data is received from the compute nodes at a server or at one of the compute nodes on which a tracing service is implemented. For example, the server may be tracing server <NUM> or <NUM>, or <NUM> shown in <FIG>, <FIG>, and <FIG>. For distributed processing employing (only) compute nodes in one or more datacenters, one of the compute nodes or another compute node or compute platform that is not being used for the distributed processing may be deployed to implemented a tracing service.

Generally, the telemetry and tracing data may be received using either a pull or push service or mechanism. For example, in one embodiment, telemetry and/or tracing data is periodically pulled from the compute nodes using a schedule or the like (although asynchronous pull modes may also be employed). In another embodiment, an agent or the like on the compute nodes may periodically or asynchronously push telemetry and/or tracing data to a platform hosting a tracing service.

In a block <NUM> the received telemetry and tracing data is aggregated using the PASIDs and GGIDs associated with those data. In an optional block <NUM>, ProcIDs and/or NodeIDs are associated with telemetry and tracing data, observing that the operations of blocks <NUM> and <NUM> may be combined. For example, in some embodiments, telemetry and/or tracing data associated with a processing element is received in a manner that associates each the ProcID for that processing element in combination with the PASID for each process and a GGID for each application for which telemetry and/or tracing data is obtained.

In a block <NUM>, a client is enabled to access aggregated telemetry and/or tracing data via the tracing service. For instance, in one embodiment tracing service is implemented as a Web service that interfaces with a client application or service running on the client using a REST API or other Web service API using JSON and/or XML data structures. As an alternative, the client functionality may be implemented on the same server or platform as the tracing service.

Under an alternatively scheme, proxy agents or the like may be used to collect telemetry and/or tracing data from compute nodes associated with the proxy agents and then forward or expose the telemetry and or tracing data. For instance, a separate proxy agent might be implemented at the data center level, at a pod level, at a rack level, or at a chassis or drawer level. Proxy agents may also be configured to perform certain levels of aggregation.

In general, the circuitry, logic and components depicted in the figures herein may also be implemented in various types of integrated circuits (e.g., semiconductor chips) and modules, including discrete chips, SoCs, multi-chip modules, and networking/link interface chips including support for multiple network interfaces. Also, as used herein, circuitry and logic to effect various operations may be implemented via one or more types of hardware-based logic, such as embedded logic, embedded processors, controllers, microengines, or otherwise using any combination of firmware executing on a processing element on the hardware. For example, the operations depicted by various logic blocks and/or circuitry may be effected using programmed logic gates and the like, including but not limited to ASICs, FPGAs, IP block libraries, or through one or more of software or firmware instructions executed on one or more processing elements including processors, processor cores, controllers, microcontrollers, microengines, etc. As used herein, the terminology "hardware-based logic" explicitly excludes software executing on or requiring use of an operating system or software running on a virtualization layer.

Although some embodiments have been described in reference to particular implementations, other implementations are possible according to some embodiments. Additionally, the arrangement and/or order of elements or other features illustrated in the drawings and/or described herein need not be arranged in the particular way illustrated and described. Many other arrangements are possible according to some embodiments.

In each system shown in a figure, the elements in some cases may each have a same reference number or a different reference number to suggest that the elements represented could be different and/or similar. However, an element may be flexible enough to have different implementations and work with some or all of the systems shown or described herein. The various elements shown in the figures may be the same or different. Which one is referred to as a first element and which is called a second element is arbitrary.

In the description and claims, the terms "coupled" and "connected," along with their derivatives, may be used. Rather, in particular embodiments, "connected" may be used to indicate that two or more elements are in direct physical or electrical contact with each other. "Coupled" may mean that two or more elements are in direct physical or electrical contact. However, "coupled" may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. Additionally, "communicatively coupled" means that two or more elements that may or may not be in direct contact with each other, are enabled to communicate with each other. For example, if component A is connected to component B, which in turn is connected to component C, component A may be communicatively coupled to component C using component B as an intermediary component.

An embodiment is an implementation or example of the inventions. Reference in the specification to "an embodiment," "one embodiment," "some embodiments," or "other embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiments is included in at least some embodiments, but not necessarily all embodiments, of the inventions. The various appearances "an embodiment," "one embodiment," or "some embodiments" are not necessarily all referring to the same embodiments.

Not all components, features, structures, characteristics, etc. described and illustrated herein need be included in a particular embodiment or embodiments. If the specification states a component, feature, structure, or characteristic "may", "might", "can" or "could" be included, for example, that particular component, feature, structure, or characteristic is not required to be included. If the specification or claim refers to "a" or "an" element, that does not mean there is only one of the element.

Italicized letters, such as 'n' and 'M' in the foregoing detailed description are used to depict an integer number, and the use of a particular letter is not limited to particular embodiments. Moreover, the same letter may be used in separate claims to represent separate integer numbers, or different letters may be used. In addition, use of a particular letter in the detailed description may or may not match the letter used in a claim that pertains to the same subject matter in the detailed description.

As discussed above, various aspects of the embodiments herein may be facilitated by corresponding firmware components, such as firmware executed by an embedded processor or the like. Thus, embodiments of this invention may be used as or to support firmware executed upon some form of processor, processing core or embedded logic or otherwise implemented or realized upon or within a non-transitory computer-readable or machine-readable storage medium. A non-transitory computer-readable or machine-readable storage medium includes any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer). For example, a non-transitory computer-readable or machine-readable storage medium includes any mechanism that provides (i.e., stores and/or transmits) information in a form accessible by a computer or computing machine (e.g., computing device, electronic system, etc.), such as recordable/non-recordable media (e.g., read only memory.

(ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, etc.). The content may be directly executable ("object" or "executable" form), source code, or difference code ("delta" or "patch" code). A non-transitory computer-readable or machine-readable storage medium may also include a storage or database from which content can be downloaded. The non-transitory computer-readable or machine-readable storage medium may also include a device or product having content stored thereon at a time of sale or delivery. Thus, delivering a device with stored content, or offering content for download over a communication medium may be understood as providing an article of manufacture comprising a non-transitory computer-readable or machine-readable storage medium with such content described herein.

The operations and functions performed by various components described herein may be implemented by firmware running on a processing element, via embedded hardware or the like, or a combination of hardware and firmware. Such components may be implemented as firmware modules, hardware modules, special-purpose hardware (e.g., application specific hardware, ASICs, DSPs, etc.), embedded controllers, hardwired circuitry, hardware logic, programmable logic, etc. Firmware content (e.g., data, instructions, configuration information, etc.) may be provided via an article of manufacture including non-transitory computer-readable or machine-readable storage medium, which provides content that represents instructions that can be executed. The content may result in a processor and/or compute node performing various functions/operations described herein.

As used herein, a list of items joined by the term "at least one of" can mean any combination of the listed terms. For example, the phrase "at least one of A, B or C" can mean A; B; C; A and B; A and C; B and C; or A, Band C.

While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the invention, as those skilled in the relevant art will recognize.

Claim 1:
A method (<NUM>, <NUM>, <NUM>) for obtaining telemetry data and tracing data for one or more applications implemented via execution of a plurality of distributed processes on a plurality of compute nodes, comprising:
for at least one of the one or more applications,
implementing (<NUM>) a time clock synchronization mechanism across a set of compute nodes executing processes associated with a first application; and
identifying (<NUM>) processes associated with the at least one application and one or more compute nodes used to execute the processes based on process application space identifiers, PASIDs, associated with the processes and based on an application identifier, ID, associated for processes associated with the at least one application; and
mapping (<NUM>) the identified processes to the application ID of the at least one application:
and
obtaining (<NUM>) at least one of telemetry data and tracing data from one or more compute nodes on which processes associated with the at least one application are executed,
wherein the at least one of telemetry data and tracing data includes the application ID for the at least one application and PASIDs for processes associated with the at least one application,
obtaining (<NUM>) time stamps associated with at least a portion of telemetry data and tracing data generated on compute nodes in the set of compute nodes using the time clock synchronization mechanism.