Patent Description:
In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one having ordinary skill in the art should recognize that the various implementations may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. For example, the dimensions of some of the elements may be exaggerated relative to other elements.

Various systems, apparatuses, and methods for determining data placement based on packet metadata are disclosed herein. In one implementation, a system includes a traffic analyzer that determines data placement across connected devices based on observed values of the metadata fields in actively exchanged packets across a plurality of protocol types. In one implementation, the protocol that is supported by the system is the compute express link (CXL) protocol. In another implementation, the system supports the Gen-Z protocol. In a further implementation, the system supports the Slingshot interconnect protocol. In other implementations, other types of interconnect protocols can be supported by the system. In one implementation, the traffic analyzer has an associated direct memory access (DMA) engine for moving data across the devices connected within the system. In some cases, the system includes multiple traffic analyzers located at various locations throughout the interconnect fabric of the system.

In one implementation, the traffic analyzer performs various actions in response to events observed in a packet stream that match items from a pre-configured list. In some cases, the pre-configured list is programmable such that the list can be updated by software. In one implementation, addresses sent on the interconnect fabric are host physical addresses (HPA's). In this implementation, the data movement is handled underneath the software applications by changing the virtual-to-physical translation once the data movement is completed. After the data movement is finished, the threads will pull in the new HPA into their translation lookaside buffers (TLBs) via a page table walker or via an address translation service (ATS) request. The traffic analyzer maintains a list of accessible media devices to which data can be relocated. In one implementation, the list of accessible media devices is updated in response to discovery events being performed.

Referring now to <FIG>, a block diagram of one implementation of a computing system <NUM> is shown. In one implementation, computing system <NUM> includes at least processors 105A-N, input/output (I/O) interfaces <NUM>, bus <NUM>, memory controller(s) <NUM>, network interface <NUM>, memory device(s) <NUM>, display controller <NUM>, display <NUM>, and devices 160A-N. In other implementations, computing system <NUM> includes other components and/or computing system <NUM> is arranged differently. The components of system <NUM> are connected together via bus <NUM> which is representative of any number and type of interconnects, links, fabric units, buses, and other connectivity modules.

Processors 105A-N are representative of any number of processors which are included in system <NUM>. In one implementation, processor 105A is a general-purpose processor, such as a central processing unit (CPU). In this implementation, processor 105A executes a driver <NUM> (e.g., graphics driver) for controlling the operation of one or more of the other processors in system <NUM>. It is noted that depending on the implementation, driver <NUM> can be implemented using any suitable combination of hardware, software, and/or firmware.

In one implementation, processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and so forth. In some implementations, processors 105A-N include multiple data parallel processors. In one implementation, processor 105N is a GPU which provides pixels to display controller <NUM> to be driven to display <NUM>. In one implementation, devices 160A-N include any number and type of accelerator devices, I/O devices, and other devices. In one implementation, each device 160A-N has its own memory which is specific to the device 160A-N. In some cases, one or more of devices 160A-N can be a processor such as a GPU.

Memory controller(s) <NUM> are representative of any number and type of memory controllers accessible by processors 105A-N and/or devices 160A-N. While memory controller(s) <NUM> are shown as being separate from processor 105A-N and device 160A-N, it should be understood that this merely represents one possible implementation. In other implementations, a memory controller <NUM> can be embedded within one or more of processors 105A-N and/or a memory controller <NUM> can be located on the same semiconductor die as one or more of processors 105A-N. Also, in one implementation, a memory controller <NUM> can be embedded within or located on the same die as one or more of devices 160A-N to access memory device(s) <NUM> that are local to devices 160A-N.

In one implementation, each memory controller <NUM> includes a traffic analyzer <NUM>. In other implementations, traffic analyzer <NUM> can be located elsewhere in system <NUM> in locations suitable for snooping traffic traversing bus <NUM>. It is noted that traffic analyzer <NUM> can also be referred to as controller <NUM>. In one implementation, traffic analyzer <NUM> performs various actions in response to events observed in a packet stream that match items from a pre-configured list. In some cases, the pre-configured list is programmable such that the list can be updated by software. In one implementation, addresses sent on the bus <NUM> are host physical addresses (HPA's). In this implementation, the data movements are handled underneath of applications by changing the virtual-to-physical translation once the data movements are completed. After the data movement is finished, the threads executing on processors 105A-N or devices 160A-N will pull in the new HPA into their translation lookaside buffers (TLBs) via a page table walker or via an address translation service (ATS) request.

Memory controller(s) <NUM> are coupled to any number and type of memory devices(s) <NUM>. For example, the type of memory in memory device(s) <NUM> includes high-bandwidth memory (HBM), non-volatile memory (NVM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), NAND flash memory, NOR flash memory, Ferroelectric Random Access Memory (FeRAM), or others.

I/O interfaces <NUM> are representative of any number and type of I/O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-Extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices (not shown) are coupled to I/O interfaces <NUM>. Such peripheral devices include (but are not limited to) displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and so forth. Network interface <NUM> is used to receive and send network messages across a network (not shown).

In various implementations, computing system <NUM> is a computer, laptop, mobile device, game console, server, streaming device, wearable device, or any of various other types of computing systems or devices. It is noted that the number of components of computing system <NUM> varies from implementation to implementation. For example, in other implementations, there are more or fewer of each component than the number shown in <FIG>. It is also noted that in other implementations, computing system <NUM> includes other components not shown in <FIG>. Additionally, in other implementations, computing system <NUM> is structured in other ways than shown in <FIG>.

Turning now to <FIG>, a block diagram of another implementation of a computing system <NUM> is shown. In one implementation, system <NUM> includes GPU <NUM>, system memory <NUM>, and local memory <NUM>. System <NUM> can also include other components which are not shown to avoid obscuring the figure. GPU <NUM> includes at least command processor <NUM>, control logic <NUM>, dispatch unit <NUM>, compute units 255A-N, DMA engine <NUM>, memory controller <NUM>, global data share <NUM>, level one (L1) cache <NUM>, and level two (L2) cache <NUM>. In one implementation, memory controller <NUM> includes traffic analyzer <NUM> for monitoring packets that traverse system <NUM>. In other implementations, GPU <NUM> includes other components, omits one or more of the illustrated components, has multiple instances of a component even if only one instance is shown in <FIG>, and/or is organized in other suitable manners. In one implementation, the circuitry of GPU <NUM> is included in processor 105N (of <FIG>).

In various implementations, computing system <NUM> executes any of various types of software applications. As part of executing a given software application, a host CPU (not shown) of computing system <NUM> launches work to be performed on GPU <NUM>. In one implementation, command processor <NUM> receives kernels from the host CPU and uses dispatch unit <NUM> to issue corresponding wavefronts to compute units 255A-N. Wavefronts executing on compute units 255A-N read and write data to global data share <NUM>, L1 cache <NUM>, and L2 cache <NUM> within GPU <NUM>. Although not shown in <FIG>, in one implementation, compute units 255A-N also include one or more caches and/or local memories within each compute unit 255A-N.

In one implementation, each compute unit 255A-N is a Single Instruction Multiple Data (SIMD) processing core. As referred to herein, a "compute unit" is a pipeline, or programming model, where respective instantiations of the same kernel are executed concurrently. Each processing element in a compute unit executes a respective instantiation of the same kernel. An instantiation of a kernel, along with its associated data, is called a work-item or thread. Thus, a kernel is the code for a work-item, and a work-item is the basic unit of work on a compute unit. All instantiations of a kernel executing on compute units 255A-N comprise a global domain of work-items. This global domain of work-items can represent the entire computation domain, and a work-item within the computation domain represents a particular task to be performed. In order to simplify execution of work-items on GPU <NUM>, work-items are grouped together into wavefronts. A wavefront is a collection of related work-items that execute together on a single compute unit.

In parallel with command processor <NUM> launching wavefronts on compute units 255A-N, DMA engine <NUM> performs various DMA operations in collaboration with traffic analyzer <NUM>. It is noted that DMA engine <NUM> is representative of any number and type of DMA engines. In one implementation, traffic analyzer <NUM> programs DMA engine <NUM> to move data between devices in response to events observed in a packet stream that match items from a pre-configured list.

Referring now to <FIG>, a block diagram of one implementation of a computing system <NUM> employing a CXL protocol is shown. In one implementation, computing system <NUM> includes host processor <NUM> and accelerator device <NUM> connected via link <NUM>. In one implementation, host processor <NUM> and accelerator device <NUM> operate in the same virtual address space. In other words, in this implementation, host processor <NUM> and accelerator device <NUM> have a unified address space. For any other processing devices, accelerators, or other devices in system <NUM>, although not shown in <FIG>, these other devices can also operate in the same virtual address space as host processor <NUM> and accelerator device <NUM>. In one implementation, system <NUM> utilizes this shared address space in compliance with the heterogeneous system architecture (HSA) industry standard.

In the example illustrated in <FIG>, host processor <NUM> includes one or more processor cores and one or more I/O devices. Host processor <NUM> also includes coherence and cache logic as well as I/O logic. In one implementation, host processor <NUM> is a CPU, while in other implementations, host processor <NUM> is any of various other types of processing units. System memory <NUM> can be included on the same package or die as host processor <NUM>. It is noted that one or more other devices can also be connected to link <NUM> although this is not shown in <FIG> to avoid obscuring the figure.

In one implementation, communication between host processor <NUM> and accelerator device <NUM> over link <NUM> can be performed according to the CXL protocol. CXL technology enables memory coherency between the memory space of host processor <NUM> and the memory of accelerator device <NUM>, allowing for higher performance and reduced complexity for applications executing on system <NUM>. The CXL traffic has three different types of traffic classes which are cache, memory, and I/O traffic classes. These traffic classes (i.e., protocol types) are multiplexed together over link <NUM> and then split out into the individual classes internally within host processor <NUM> and accelerator device <NUM>. While system <NUM> is shown and described as employing the CXL protocol, it should be understood that this is merely indicative of one particular implementation. Other types of systems employing other types of coherent interconnect protocols can also take advantage of the techniques presented herein.

In one implementation, host processor <NUM> includes a traffic analyzer <NUM> and accelerator device <NUM> includes a traffic analyzer <NUM>. Traffic analyzers <NUM> and <NUM> monitor the packets that are sent over link <NUM>. Traffic analyzers <NUM> and <NUM> analyze the metadata associated with the packets to determine how to move data between system memory <NUM> and device-attached memory <NUM>. In one implementation, each traffic analyzer <NUM> and <NUM> includes a list of events which determine when the movement of data is triggered. If the metadata in a given packet meets the criteria in one of the entries of the list of events, then the traffic analyzer causes data to be moved to the memory which is closest to the device which is more likely to use the data in the near future. While two traffic analyzers <NUM> and <NUM> are shown in <FIG>, it should be understood that other implementations can have other numbers of traffic analyzers (e.g., <NUM>, <NUM>, <NUM>).

Turning now to <FIG>, a block diagram of one implementation of a controller <NUM> is shown. It is noted that controller <NUM> can also be referred to as traffic analyzer <NUM>. In one implementation, traffic analyzer <NUM> (of <FIG>), traffic analyzer <NUM> (of <FIG>), and traffic analyzers <NUM> and <NUM> (of <FIG>) include the components and functionality of controller <NUM>. Control unit <NUM> is connected to DMA engine <NUM> and uses DMA engine <NUM> to move data across the devices that are included within the overall system (e.g., system <NUM> of <FIG>). Control unit <NUM> can be implemented using any combination of circuitry, execution units, processor cores, memory elements, and/or program instructions.

Controller <NUM> maintains and/or accesses accessible media device list <NUM> and event list <NUM> during operation. Accessible media device list <NUM> includes identifications of media device(s) that are accessible by controller <NUM>. Accessible media device list <NUM> is updated upon discovery events, vendor defined messages (VDMs), and based on other operations. Event list <NUM> includes a listing of events that trigger data movement. In one implementation, the lowest priority events are listed first in event list <NUM>. In another implementation, each event has an associated priority indicator which is stored with the event. If two events are observed simultaneously and trigger conflicting actions, then the higher priority event is satisfied first.

For example, if a first event commands a prefetcher on a given core or device to be enabled and a second event causes the prefetcher on the given core or device to be disabled, then the higher priority event will be performed first. Also, if a first event requests that a given data region be moved to a first location, and a second event requests to move the given data region to a second location, then the higher priority event will satisfied first. Still further, if two different events request a DMA engine, but the requests cannot be satisfied at the same time by the DMA engine, then the higher priority event will be performed first. In one implementation, after an event triggers a movement for a given data region, there is a period of time (i.e., a cool down period) during which no other event can trigger another movement of the given data region.

Referring now to <FIG>, one example of an event list <NUM> is shown. In one implementation event list <NUM> (of <FIG>) includes the entries of event list <NUM>. Event <NUM> refers to an I/O address translation service (ATS) translation completion event. In one implementation, an ATS translation completion restricts access to a CXL. IO protocol type by setting the "CXL" bit in the translation completion date entry. In one implementation, the "CXL" bit is set for an uncacheable type of data. When such an event occurs, the controller places the corresponding page on a media that is suited for I/O style, block accesses. This is due to the fact that cache line granularity CXL. cache and CXL. mem accesses will likely not happen to the page in the near future. In one implementation, a media that is suited for I/O style, block accesses is a flash device. As used herein, the term "flash" device refers to an electronic non-volatile computer memory storage medium that can be written to or erased with electricity. There are two main types of flash memory, which are NOR flash and NAND flash. In one implementation, entry <NUM> has a medium priority.

Entry <NUM> refers to a bias flip request event. Bias flip requests are sent from the device to the host to invalidate a cache line from the host's caches. In one implementation, bias flip requests are sent on the CXL. cache request channel using the RdOwnNoData opcode. A bias flip request is a strong indication that the memory will be used by the device in the near future. As a consequence, the controller will pull the memory region associated with the bias flip request to the device's local memory. In one implementation, bias flip requests have a high priority.

Entry <NUM> refers to an event when "N" read for ownership packets to cache lines of the same page are detected, where "N" is a positive integer. In one implementation, the size of a cache line is <NUM> bytes and the size of a page is <NUM> kilobytes (KB). However, the sizes of cache lines and pages can vary in other implementations. Also, the value of "N" can vary from implementation to implementation, with "N" serving as a threshold for triggering data movement. A read for ownership packet refers to an operation where a device is caching data in any writeable state (e.g., modified state, exclusive state). The action performed by the controller in response to detecting this event is to pull the data to the local device memory due to the likelihood of subsequent memory accesses to the same data range. In one implementation, entry <NUM> has a high priority.

Entry <NUM> refers to an atomic write of a full cache line. When "N" of these packets to the same page are detected, the data will be pulled to the local device memory. If the request misses in the last level cache, the data will be written to memory. In one implementation, entry <NUM> has a high priority.

Entry <NUM> refers to a clean eviction of a cache line. In one implementation, entry <NUM> occurs when the host generates a request to evict data from a device cache. Entry <NUM> can refer to a clean evict request with data or a clean evict without data. In other words, the host either asks for the clean data or not. The action performed in response to detecting "N" of these requests to the same page is to pull the data to the main memory of the host. In one implementation, entry <NUM> has a medium priority.

Entry <NUM> refers to an eviction of modified data from a device cache which is requested by the host. If "N" of these dirty evict packets are detected to the same page, then the action performed is to transfer the corresponding modified data to the host main memory. In one implementation, entry <NUM> has a high priority.

Entry <NUM> refers to a snoop request when the host will cache the data in a shared or exclusive state. The device will degrade the cache line to shared or invalid and return dirty data. If "N" of these snoop request packets are detected to the same page, then the action performed is to transfer the page to the host main memory. In one implementation, entry <NUM> has a medium priority.

Entry <NUM> refers to a snoop invalidate request when the host will cache the data in an exclusive state. The device will invalidate the cache line in response to receiving the snoop invalidate request. If "N" of these snoop invalidate request packets are detected to the same page, then the action performed is to transfer the page to the host main memory. In one implementation, entry <NUM> has a medium priority.

Entry <NUM> refers to a memory read packet when the host wants an exclusive copy of the cache line. When this event is detected, the action taken is to pull the corresponding data to the host main memory for subsequent accesses. In one implementation, entry <NUM> has a high priority.

Entry <NUM> refers to a memory read packet when the host wants a non-cacheable but current copy of the cache line. When this event is detected, the action taken is to pull the corresponding data to the host main memory for subsequent accesses. In one implementation, entry <NUM> has a medium priority.

It should be understood that the example of entries <NUM>-<NUM> shown in event list <NUM> are merely indicative of one particular implementation. In other implementations, event list <NUM> can have other numbers and/or types of entries corresponding to other types of events which will trigger data movement. It is also noted that the value of "N" can vary from entry to entry, with one entry having a higher value of "N" and another entry having a lower value of "N". For example, one entry can cause a movement of a page when <NUM> packets targeting the page are detected while another entry can cause a movement of a page when <NUM> packets associated with cache lines of the page are detected. In other words, the threshold for data movement depends on the event type, with different events having different thresholds.

While the priorities of entries <NUM>-<NUM> are shown as taking on values of High, Medium, and Low, this is merely representative of one implementation. These values can be converted into numeric or binary values in other implementations. The number of bits that are used to encode the priority values can vary according to the implementation. Also, while a single list <NUM> is shown in <FIG>, it should be understood that list <NUM> is representative of any number of lists that can be maintained and accessed by a traffic analyzer. For example, in another implementation, the traffic analyzer maintains three lists, one for the CXL. memory protocol type, one for the CXL. cache protocol type, and one for the CXL. io protocol type. Depending on the type of packet that is detected, the traffic analyzer will query the list corresponding to the packet's protocol type.

Turning now to <FIG>, one implementation of a method <NUM> for determining data placement based on packet metadata is shown. For purposes of discussion, the steps in this implementation are shown in sequential order. However, it is noted that in various implementations of the described methods, one or more of the elements described are performed concurrently, in a different order than shown, or are omitted entirely. Other additional elements are also performed as desired. Any of the various systems or apparatuses described herein are configured to implement method <NUM>.

A traffic analyzer maintains a list of events for triggering data movement in a computing system (block <NUM>). Also, the traffic analyzer monitors a stream of packets traversing a link (block <NUM>). The link can be an interconnect, a fabric, a bus, a memory channel, or otherwise. For each packet traversing the link, the traffic analyzer examines metadata associated with the packet (block <NUM>). In another implementation, the traffic analyzer analyzes a portion (i.e., subset) of the packets traversing the link. For example, in one implementation, the traffic analyzer examines one of every four packets traversing the link. Other ratios (e.g., <NUM>/<NUM>, <NUM>/<NUM>) are possible in other implementations. The traffic analyzer determines if the metadata indicates that the packet meets the criteria specified in any event entry of the list of events (block <NUM>).

If metadata indicates that a given packet meets the criteria specified in any event entry of the list of events (conditional block <NUM>, "yes" leg), then the traffic analyzer causes a corresponding block of data to be moved from a first memory device to a second memory device (block <NUM>). It is assumed for the purposes of this discussion that the second memory device is different from the first memory device. In one implementation, the traffic analyzer queries an accessible media device list (e.g., accessible media device list <NUM> of <FIG>) to determine a preferred memory device for migrating the corresponding block of data. In one implementation, the traffic analyzer is coupled to a DMA engine, and the traffic analyzer programs the DMA engine to perform the data movement from the first memory device to the second memory device.

Also, the virtual-to-physical address translation is updated for the block of data once the data movement is complete (block <NUM>). Updating the translation can involve any number of actions being performed in a particular order, such as notifying the operating system (OS), pausing an application, initiating a TLB shootdown, and so on. Depending on the implementation and the system architecture, the traffic analyzer can be programmed to perform these and/or other steps in an order specific to the system architecture. In some cases, the traffic analyzer performs steps to update the translation in the page table and TLBs without invoking the OS. In other cases, the traffic analyzer cooperates with the OS so as to update the translation. For example, in one implementation, the traffic analyzer sends a packet over the link to the OS requesting the translation update. After block <NUM>, method <NUM> returns to block <NUM> with the traffic analyzer continuing to monitor the stream of packets traversing the link.

Otherwise, if the metadata of the given packet does not meet the criteria specified in any event entry of the list of events (conditional block <NUM>, "no" leg), then method <NUM> returns to block <NUM> with the traffic analyzer continuing to monitor the stream of packets traversing the link.

In various implementations, program instructions of a software application are used to implement the methods and/or mechanisms described herein. For example, program instructions executable by a general or special purpose processor are contemplated. In various implementations, such program instructions are represented by a high level programming language. In other implementations, the program instructions are compiled from a high level programming language to a binary, intermediate, or other form. Alternatively-, program instructions are written that describe the behavior or design of hardware. Such program instructions are represented by a high-level programming language, such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various implementations, the program instructions are stored on any of a variety of non-transitory computer readable storage mediums. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

Claim 1:
An apparatus comprising:
a control unit configured to:
monitor a stream of packets traversing a link (<NUM>); and
responsive to determining a packet of the stream of packets meets criteria associated with a first event (<NUM>):
cause a corresponding block of data to be moved from a first memory device to a second memory device (<NUM>); and
update a virtual-to-physical address translation for the block of data (<NUM>).