Patent Description:
In some processing systems, a computing workload may be split among multiple compute devices, each of which may include a processor and memory. Data produced as a result of a first computation by a first one of the compute devices may be stored at a storage device, then transferred to a second one of the compute devices where it may be used as an input to a second computation. A host device may coordinate data movement between the compute devices and the storage device.

The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not constitute prior art.

From <CIT> there is known an apparatus for implementing a software-transparent hardware predictor for core-to-core data communication optimization are described herein. The apparatus includes a plurality of hardware processor cores each including a private cache; a shared cache that is communicatively coupled to and shared by the plurality of hardware processor cores; and a predictor circuit. The predictor circuit is to track activities relating to a plurality of monitored cache lines in the private cache of a producer hardware processor core (producer core) and to enable a cache line push operation upon determining a target hardware processor core (target core) based on the tracked activities.

From <CIT> there is known a storage device that includes a storage region in which first data is stored and that is accessed using a first virtual address, and a memory controller configured to control stored data stored in the storage region. The memory controller predicts second data to be accessed using a second virtual address based on the first virtual address, prefetches the second data into an external device, and modifies a physical address mapped to the second virtual address so that the prefetched second data is accessible by a host in communication with the storage device.

From <CIT> and <CIT> there are known direct cache transfer (DCT) mechanisms that initiate a DCT at the time the updated data is being evicted from the producer cache. Since the direct cache transfer takes place at the time the updated data is being evicted, by the time the consumer begins its task, the updated contents have already been placed in its own cache or another cache within the cache hierarchy.

<CIT> discloses a producer/consumer data processing model for data migration techniques / external cache allocation ("cache push" and prefetch); uses prefetch hints, originating from the consumer.

<CIT> discloses data buffering in media and pipelined processing components, wherein a buffer (i.e. first fill last spills buffer) has a consumer reading information corresponding to a data structure from a memory and processing the data from the producer based on the data structure.

<CIT> discloses a distributed neural network operating on a producer/consumer data processing model.

The figures are not necessarily drawn to scale and elements of similar structures or functions may generally be represented by like reference numerals or portions thereof for illustrative purposes throughout the figures. The figures are only intended to facilitate the description of the various embodiments described herein. The figures do not describe every aspect of the teachings disclosed herein and do not limit the scope of the claims. To prevent the drawings from becoming obscured, not all of the components, connections, and the like may be shown, and not all of the components may have reference numbers. However, patterns of component configurations may be readily apparent from the drawings. The accompanying drawings, together with the specification, illustrate example embodiments of the present invention, and, together with the description, serve to explain the principles of the present invention defined by the appended claims.

A storage device in accordance with example embodiments of the invention may prefetch data stored at the storage device and transfer it to a consumer device that may use the data for a computation or other processing. In some embodiments, this may reduce or eliminate the involvement of a host which may be a bottleneck in transferring data between devices. Depending on the implementation details, prefetching data and transferring it to a consumer device may reduce access latency and/or synchronization overhead, and/or may enable data input and/or output (I/O) operations to overlap with data processing operations at the consumer device, thereby improving throughput.

In some embodiments, a producer device and a consumer device may be coupled through an interconnect in a pipeline configuration to perform distributed computations such as machine learning (ML) training and/or inference. For example, a producer device (e.g., a compute device such as an accelerator, graphics processing unit (GPU), and/or the like) may write the results of a first stage of computation to a storage device through the interconnect. A consumer device (e.g., another compute device such as an accelerator, GPU, and/or the like) may read the results from the storage device and use the results for a next stage of computation. In some embodiments, a prefetcher in the storage device may prefetch the results stored by the producer device and transfer the results to the consumer device in anticipation of the consumer device using the results for the next stage of computation. Depending on the implementation details, this may enable data to be transferred to the consumer device in parallel with other processing being performed by the consumer device, thereby reducing or hiding memory and/or storage device access latency.

A storage device may determine which consumer device to transfer prefetched data to based on various techniques in accordance with example embodiments of the invention. In an embodiment, a prefetcher for a storage device receives information from an application (e.g., running on a host coupled to the interconnect) indicating producer-consumer relationships between one or more producer devices and one or more consumer devices. Thus, when a specific producer device writes data to the storage device (e.g., a specific amount of data written to a specific location), the prefetcher may prefetch the data and transfer it to a specific consumer device. As another example, in some embodiments, a prefetcher may monitor read and/or write operations for a storage device to detect one or more access patterns that may predict which consumer device is likely to use data stored by a specific producer device.

To provide a target location for writing prefetched data at a consumer device, a storage device may allocate memory at a consumer device based on various techniques in accordance with example embodiments of the invention. For example, in some embodiments, a storage device may send a memory allocation request to a host which may allocate target memory at the consumer device (e.g., through a virtual memory manager (VMM) at the host). As another example, the storage device may allocate the target memory itself (e.g., using a VMM at the prefetcher). In some embodiments in which the storage device allocates the target memory, the storage device may copy the prefetched data to a reserved area of memory at the consumer device.

In some embodiments, an interconnect between a producer device, a consumer device, a storage device, and/or a host may be implemented at least partially with a memory coherent interface and/or using one or more memory coherent protocols. In such embodiments, one or more aspects of the memory coherent interface and/or protocol may be used to implement one or more features in accordance with example embodiments of the invention. For example, in some embodiments, a coherency engine may send information about one or more producer-consumer relationships to a prefetcher using one or more protocol fields such as a tag field.

In some embodiments, a storage device may store data from one or more producer devices in one or more streams at the storage device. For example, data having similar lifetimes and/or similar producer-consumer relationships may be placed in the same streams. Thus, in some embodiments, data destined for the same consumer device may be placed in the same stream. Depending on the implementation details, this may improve garbage collection and/or block erase operations at the storage device, because, for example, some or all of the data transferred to a specific consumer device may become invalid at the same time.

<FIG> illustrates an embodiment of a system for splitting a processing workload among multiple compute devices in accordance with example embodiments of the invention. The system illustrated in <FIG> may include a host device <NUM>, four compute devices 104a, 104b, 104c, and 104d (which may be referred to collectively as <NUM>), and two storage devices 106a and 106b (which may be referred to collectively as <NUM>). The host device <NUM>, compute devices <NUM>, and storage devices <NUM> may communicate through an interconnect <NUM>. Each of the compute devices <NUM> may process a corresponding stage of an ML workload <NUM>, which in this embodiment, may be implemented as a neural network. Thus, compute devices 104a, 104b, 104c, and 104d may process corresponding stages 110a, 110b, 110c, and 110d, respectively, of the neural network workload <NUM>. The final stage 110d may include, for example, one or more fully connected (FC) layers and a SoftMax function.

The host device <NUM> may include a central processing unit (CPU) <NUM> and a memory <NUM> which, in this embodiment, may be implemented with dynamic random access memory (DRAM). Each of the compute devices 104a, 104b, 104c, and 104d may include a corresponding GPU 116a, 116b, 116c, and 116d, respectively (indicated as GPU0, GPU1, GPU2, and GPU3, respectively). The GPUs 116a, 116b, 116c, and 116d may be referred to collectively as <NUM>. Each of the compute devices 104a, 104b, 104c, and 104d may further include a corresponding local device memory 118a, 118b, 118c, and 118d, respectively (indicated as DRAM0, DRAM1, DRAM2, and DRAM3, respectively). The local device memories 118a, 118b, 118c, and 118d may be d to collectively as <NUM>. Each of the storage devices 106a and 106b may include a corresponding local storage medium 120a and 120b, respectively (indicated as Storage0 and Storage1, respectively). The local storage medium 120a and 120b may be referred to collectively as <NUM>. Each of the storage devices 106a and 106b may further include a corresponding controller 122a and 122b, respectively, (indicated as Controller0 and Controller1, respectively). The controllers 122a and 122b may be referred to collectively as <NUM>.

In some embodiments, an application running on the host device <NUM> may coordinate data movement between the individual device local memories. For example, the host device <NUM> may send one or more commands to one of the storage devices <NUM> to transfer data from the local memory <NUM> of one of the compute devices 104a-d to the storage medium <NUM> of the storage device <NUM>. This may be referred to as pulling data from the local memory <NUM>. The host device <NUM> may also send one or more commands to one of the storage devices <NUM> to transfer data from the storage medium <NUM> of the storage device <NUM> to the local memory <NUM> of one of the compute devices 104a-d. This may be referred to as pushing data to the local memory <NUM>.

In the embodiment illustrated in <FIG>, an example data flow coordinated by the CPU <NUM> of host device <NUM> is shown by data transfers (<NUM>), (<NUM>), (<NUM>), (<NUM>), (<NUM>), (<NUM>), and (<NUM>). Thus, at operation (<NUM>), first data may first be pushed from Storage0 to DRAM0 where it may be read and used as an input to a computation performed by GPUO. At operation (<NUM>), second data may be pushed from Storage0 to DRAM1. However, a computation using the second data at GPU1 may wait until a result of the computation performed by GPUO is stored as third data in DRAM0 then transferred at operation (<NUM>) to DRAM1. The second and third data may be used as inputs to a computation performed by GPU1, the result of which may be written as fourth data to DRAM1. The fourth data may then be pulled to Storage1 at operation (<NUM>). Fifth data may be pushed from Storage1 to DRAM2 at operation (<NUM>). The fifth data may be used as an input to a computation by GPU2, the output of which may be written as sixth data to DRAM2. The sixth data may be transferred to DRAM3 at operation (<NUM>) then used as an input to a computation performed by GPU3, the output of which may be written as seventh data to DRAM3. The seventh data may then be pulled to Storage1 at operation (<NUM>).

Depending on the implementation details, the host device <NUM> may be a bottleneck for data movement between devices because it may be involved in coordinating some or all of the data transfers. Thus, the storage devices <NUM> may be passive participants in the data movement. Moreover, in some embodiments, data transfers between the local memories <NUM> and the storage media <NUM> may only occur while a processing kernel is not executing on the corresponding GPU <NUM>.

<FIG> illustrates an embodiment of a system with data prefetching and transfer in accordance with example embodiments of the invention. The system illustrated in <FIG> may include a first compute device 204a, a second compute device 204b, a storage device <NUM>, and a prefetcher <NUM>, all of which may communicate through an interconnect <NUM>. The first and second compute devices 204a and 204b may each include a corresponding processor or other general initiator (GI) 216a and 216b, respectively, and a corresponding memory 218a and 218b, respectively. The storage device <NUM> may include a storage medium <NUM>.

In some embodiments, one or more of the compute devices <NUM> may operate as a producer device that may produce (e.g., as a result of a computation or other processing) data that may be consumed by one or more of the compute devices <NUM> that may operate as a consumer device. In some situations, a compute device <NUM> may operate as both a producer device and a consumer device.

The prefetcher <NUM> may implement one or more techniques for storing and/or transferring data to and/or from one or more of the compute devices <NUM> and/or other devices accessible through the interconnect <NUM> in accordance with example embodiments of the invention. For example, the prefetcher <NUM> may be implemented as a programmable prefetcher that may prefetch data from local memory at the storage device <NUM> (e.g., storage medium <NUM>) and push it to the local memory <NUM> of one or more of the compute devices <NUM> (e.g., a memory at the device having a processor or other GI <NUM> that may use the data, or a memory at a device that may be relatively close, or closest, to a processor or other GI that may use the data. Thus, in some embodiments, a consumer device may be a compute device <NUM> that may include a processor or other GI that may use the transferred data, or a consumer device may be a compute device <NUM> or other device having a memory that may store the transferred data for a processor or other Gl (e.g., at another device connected to the interconnect <NUM>) that may use the transferred data.

In some embodiments, the prefetcher <NUM> may determine a consumer device to prefetch data for, and/or push data to, based on information the prefetcher may receive from an application (e.g., running on a host coupled to the interconnect) indicating one or more producer-consumer relationships between one or more producer devices and one or more consumer devices. In some embodiments, the prefetcher <NUM> may determine a consumer device by monitoring one or more read and/or write operations for one or more storage devices to detect one or more access patterns that may predict which consumer device is likely to use data stored by a specific producer device. In some embodiments, the prefetcher <NUM> may include detection logic <NUM> configured to monitor read and/or write operations and/or detect one or more access patterns.

In some embodiments, the prefetcher <NUM> may allocate memory at a consumer device by requesting a memory allocation by a host device, by allocating the memory itself, or in any other manner.

Depending on the implementation details, the embodiment illustrated in <FIG> may reduce, eliminate, and/or hide memory and/or storage access latency for one or more compute devices, storage devices and/or other devices accessible through the interconnect <NUM>. This may reduce or eliminate reliance on a host and/or CPU to coordinate data movement, which in turn, may result in lower CPU utilization. Moreover, depending on the implementation details, data transfers to and/or from consumer and/or producer devices may overlap with other processing (e.g., kernel execution) at the consumer and/or producer devices, thereby improving throughput.

In some embodiments, the prefetcher <NUM> may be integral with the storage device <NUM>. For example, in some embodiments the prefetcher may be implemented partially or entirely as part of a storage device controller for the storage device <NUM>. As another example, in some embodiments, the prefetcher <NUM> may be implemented partially or entirely as part of a host device and/or one or more of the compute devices <NUM>.

The compute devices <NUM> may be implemented with any type of device that may include memory <NUM> and/or processor or other Gl <NUM> that may produce and/or use data that may be stored in the storage device <NUM>. Examples may include GPUs, accelerators, neural processing units (NPUs), tensor processing units (TPUs), network interface cards (NICs), and/or the like.

Any of the memories 218a and 218b and/or storage medium <NUM> may be implemented with any type of memory and/or storage media including any type of solid state media, magnetic media, optical media, and/or the like, any type of volatile memory such DRAM, static random access memory (SRAM), and/or the like, any type of nonvolatile memory including flash memory such as not-AND (NAND) flash memory, persistent memory (PMEM) such as cross-gridded nonvolatile memory, memory with bulk resistance change, phase change memory (PCM), and/or the like, or any combination thereof.

The interconnect <NUM> may be implemented one or more of any type of interface and/or protocol including Peripheral Component Interconnect Express (PCIe), Nonvolatile Memory Express (NVMe), NVMe-over-fabric (NVMe-oF), Ethernet, Transmission Control Protocol/Internet Protocol (TCP/IP), remote direct memory access (RDMA), RDMA over Converged Ethernet (ROCE), FibreChannel, InfiniBand, Serial ATA (SATA), Small Computer Systems Interface (SCSI), Serial Attached SCSI (SAS), iWARP, and/or the like, or any combination thereof. In some embodiments, the interconnect <NUM> may be implemented with one or more memory semantic and/or memory coherent interfaces and/or protocols such as Compute Express Link (CXL), and/or CXL. io, and/or CXL. cache, Gen-Z, Coherent Accelerator Processor Interface (CAPI), Cache Coherent Interconnect for Accelerators (CCIX), and/or the like, or any combination thereof.

For purposes of illustration, the embodiment illustrated in <FIG> may include a device <NUM> that is implemented as a storage device. However, the principles of this invention may be implemented with any type of device that may be used to store, prefetch, and/or transfer data in accordance with example embodiments of the invention. Examples of devices that may prefetch and transfer data may include caching devices (e.g., CXL Type-<NUM> devices), accelerators with memory (e.g., CXL Type-<NUM> Devices), memory buffer devices (e.g., CXL Type-<NUM> devices), NICs, with memory, and/or the like.

<FIG> illustrates an example embodiment of a system with data prefetching and transfer in accordance with example embodiments of the invention. The embodiment illustrated in <FIG> may be used, for example, to implement the system illustrated in <FIG> and/or any prefetching and/or data transfer features described herein.

Referring to <FIG>, the system may include a host device <NUM>, any number of (in this example, four) compute devices 304a, 304b, 304c, and 304d (which may be referred to collectively as <NUM>), and any number of (in this example, two) storage devices 306a and 306b (which may be referred to collectively as <NUM>). The host device <NUM>, compute devices <NUM>, and/or storage devices <NUM> may communicate through an interconnect <NUM>.

For purposes of illustration, each of the compute devices <NUM> may process a corresponding stage of an ML workload <NUM>, which in this embodiment, may be implemented as a neural network. Thus, compute devices 304a, 304b, 304c, and 304d may process corresponding stages <NUM>0a, <NUM>0b, 310c, and 310d, respectively, of the neural network workload <NUM>. The final stage 310d may include, for example, one or more fully connected (FC) layers and a SoftMax function. However, the system illustrated in <FIG> may be used for any other type of computations and/or processing.

The host device <NUM> may include a central processing unit (CPU) <NUM> and a memory <NUM> which, in this embodiment, may be implemented with dynamic random access memory (DRAM), but may also be implemented with any other type of memory.

For purposes of illustration, each of the compute devices 304a, 304b, 304c, and 304d may include a corresponding GPU 316a, 316b, 316c, and 316d, respectively (indicated as GPU0, GPU1, GPU2, and GPU3, respectively). The GPUs 316a, 316b, 316c, and 316d may be referred to collectively as <NUM>. However, any other type of compute and/or processing apparatus may be used.

Each of the compute devices 304a, 304b, 304c, and 304d may further include a corresponding local device memory 318a, 318b, 318c, and 318d, respectively (indicated as DRAM0, DRAM1, DRAM2, and DRAM3, respectively). The local device memories 318a, 318b, 318c, and 318d may be referred to collectively as <NUM>. For purposes of illustration, the memories <NUM> may be implemented with DRAM as shown in <FIG>, but any other type of memory may be used.

Each of the storage devices 306a and 306b may include a corresponding local storage medium 320a and 320b, respectively (indicated as Storage0 and Storage1, respectively). The local storage medium 320a and 320b may be referred to collectively as <NUM>. For purposes of illustration, the storage media <NUM> may be assumed to be NAND flash memory, but any type of memory and/or storage media may be used.

Each of the storage devices 306a and 306b may further include a corresponding prefetcher 324a and 324b, respectively, (indicated as Prefetcher0 and Prefetcher1, respectively). The prefetchers 324a and 324b may be referred to collectively as <NUM>.

For purposes of illustration, the interconnect <NUM> may be implemented with CXL, but any other type of interconnect(s) and/or protocol(s) may be used.

One or more of the CPU <NUM>, the GPUs <NUM>, and/or prefetchers <NUM> may be assigned a general initiator identifier (GI ID), for example, by the host device <NUM>. In the embodiment illustrated in <FIG>, the CPU <NUM>, GPUs 316a, 316b, 316c, and 316d and prefetchers 324a and 324b may be assigned GI ID <NUM>, GI ID <NUM>, GI ID <NUM>, GI ID <NUM>, GI ID <NUM>, GI ID <NUM>, GI ID <NUM>, respectively. The GI IDs may be used, for example, to keep track of producer-consumer relationships and/or to facilitate the transfer of data, command, and/or the like throughout the system.

Any of the prefetchers <NUM> may push data to any of the memories <NUM> and/or <NUM> using connections through the interconnect <NUM>, some examples of which are shown by dashed arrows <NUM>. Any of the prefetchers <NUM> may communicate with any of the GPUs <NUM> and or CPU <NUM> using connections through the interconnect <NUM>, some examples of which are shown by solid arrows <NUM>.

<FIG> illustrates an example embodiment of a method for storing data in accordance with example embodiments of the invention. The embodiment illustrated in <FIG> may be used, for example, with any of the systems disclosed herein, including those illustrated in <FIG> and/or <FIG>.

Referring to <FIG>, a storage device <NUM> may include a multi-stream interface <NUM>, a flash translation layer (FTL) <NUM> and a storage medium (in this example, NAND flash memory) <NUM>.

An application <NUM> running on a host device <NUM> provides one or more indications of producer-consumer relationships to a prefetcher <NUM>. The one or more indications (which may also be referred to as hints) may include information such as a producer GI ID, a consumer Gl ID, a data address, and/or a data size (in bytes, pages, blocks, and/or the like) as illustrated in Table <NUM> which may be stored by the prefetcher <NUM>.

In some embodiments, the application <NUM> may pass the producer and/or consumer GI IDs to the prefetcher, for example, during data reads and/or writes using one or more CXL fields such as a tag field and/or a metavalue field and metafield field. The host device <NUM> and/or application <NUM> may be implemented, for example, with the corresponding host device <NUM> illustrated in <FIG> as shown by arrow <NUM>. In some embodiments, the application <NUM> may provide the one or more indications of producer-consumer relationships to a prefetcher <NUM> programmatically, for example, by programming the prefetcher through an application programming interface (API). In some embodiments, the prefetcher <NUM> may further include detection logic <NUM> to monitor data reads and/or writes to detect one or more producer-consumer relationships.

Referring to <FIG>, in some embodiments, data provided by the application <NUM> and/or a producer device may be stored in one or more streams and/or blocks associate with streams in the storage medium <NUM> of a storage device based, for example, on one or more producer-consumer relationships and/or one or more data lifetimes. For example, as shown in <FIG>, data pages Data0, Data1, Data2, Data3, Data4, and/or Data5 in application <NUM> may have producer-consumer relationships and/or data lifetimes indicated by the various shading shown in <FIG>. The application <NUM> is shown providing Producer Gl ID <NUM> and Consumer GI ID <NUM> for data page Data1 to the prefetcher <NUM> as shown by arrow <NUM>. Based on producer producer-consumer relationships such as those shown in Table <NUM>, and/or data lifetimes, the prefetcher may store, through the multi-stream interface <NUM> and FTL <NUM>, data in Block0, Block1, Block2, and/or Block3 of the storage medium <NUM> associated with one or more streams identified by stream identifiers Stream ID <NUM>, Stream ID <NUM>, Stream ID <NUM>, and Stream ID <NUM>, respectively.

In the example illustrated in <FIG>, Data1 and Data5 may be placed in Block0, Data0 and Data4 may be stored in Block1, Data3 may be stored in Block <NUM>, and Data2 may be stored in Block3.

Thus, in some embodiments, a prefetcher may exploit existing apparatus for stream-based placement to place related data in the same stream, which, depending on the implementation details, may provide an efficient storage technique for data to be prefetched and/or pushed to a compute device.

<FIG> illustrates an example embodiment of a method for storing, prefetching, and transferring data in accordance with example embodiments of the invention. The embodiment illustrated in <FIG> may be used, for example, with any of the systems and/or methods disclosed herein.

Referring to <FIG>, at operation <NUM>, an application may send information including one or more indications of one or more producer-consumer relationships to a prefetcher of a storage device. The prefetcher may store the information which may include GI IDs and/or relationships, for example, in a data structure such as Table <NUM>.

At operation <NUM>, the storage device may make one or more data placement decisions (e.g., using the prefetcher) based, for example, on one or more indications from the application, for storing data at the device. For example, the prefetcher may select one or more streams for storing data received from a host and/or one or more producer devices based on one or more indications of producer-consumer relationships. At operation <NUM>, the prefetcher may then store the data in the selected streams through a multi-stream interface in the storage device.

At operation <NUM>, the storage device may detect, e.g., using detection logic in the prefetcher, one or more access patterns that may indicate a producer-consumer relationship between one or more producer devices and one or more consumer devices. The detection of access patterns may be in addition to, or an alternative to, the indications of producer-consumer relationship provided by an application and/or host. Based on one or more indicated producer-consumer relationship and/or one or more detected access patterns, the prefetcher may select one or more consumer devices to prefetch data for, and one or more times to prefetch the data. For example, the prefetcher may prefetch data for a specific consumer device when there is free space for the data in the memory of the consumer device.

At operation <NUM>, the prefetcher may push the prefetched data to the consumer device through an interconnect such as CXL. In some embodiments, the prefetcher may perform one or more operations to allocate target space for the data at the consumer device prior to pushing the data as described in more detail below.

In some embodiments, an application may provide the one or more indications of producer-consumer relationships to a prefetcher programmatically, for example, by programming the prefetcher through an application programming interface (API). Such an arrangement may be used, for example, when a user or programmer may have insights into the data access patterns of a workload. An example of a pseudocode definition for a procedure for sending one or more indications (e.g., hints) to a prefetcher may be as follows:
send_prefetch_hint (const void * prefetcher, size_t producer_id, size_t consumer_id, const void * buffer_ptr, size_t size, string access_pattern);.

Examples of parameters that may be provided with an indication of a producer-consumer relationship may be as follows:.

An example invocation of the procedure for sending one or more indications to a prefetcher is the a case in which the application provides an access pattern for the prefetcher to identify (e.g., the prefetcher may push data to GPU1 before the end of GPUO kernel execution):
send_prefetch_hint (,,,,,"sequential"), <NUM>-><NUM>.

An example invocation of the procedure for a case in which an access pattern may be determined by the prefetcher at runtime may be as follows:
send_prefetch_hint (,,,,,"runtime"), <NUM>-><NUM>-><NUM>-><NUM>.

<FIG> illustrates an example embodiment of a method for prefetching data in accordance with example embodiments of the invention. The embodiment illustrated in <FIG> may be used, for example, with any of the systems and/or methods disclosed herein, but for purposes of illustration, the embodiment illustrated in <FIG> may be described in the context of the system illustrated in <FIG>.

Referring to <FIG>, at operation (<NUM>), the GPU 316a (GPUO) writes <NUM> data elements to storage medium 320a (Storage0) as indicated by the dashed line <NUM>. To notify the prefetcher 324a (Prefetcher0) that the write is complete, GPUO may write any data to a predetermined memory location using, for example, the CXL interconnect.

In an implementation in which the prefetcher determines an access pattern at runtime, the Prefetcher0 may observe, at operation (<NUM>), that GPU1 may read data elements 640a, 640b, 640c, and 640d in sequence after GPUO writes the data <NUM>. At operation (<NUM>), based on the observed access pattern, Prefetcher0 prefetches the data <NUM> when it observes GPUO writing the data <NUM>. Alternatively, or additionally, Prefetcher0 may observe GPU1 sequentially reading data elements 640a, 640b, 640c, and 640d and therefore prefetch data elements 640e, 640f, <NUM>, and 640i on the assumption that GPU1 will read those data elements next.

In an implementation in which the prefetcher is provided a producer-consumer relationship between GPUO and GPU1, Prefetcher0 does not need to observe the operation (<NUM>) and instead, at operation (<NUM>), Prefetcher0 prefetches the data <NUM> based on the producer-consumer relationship when GPUO writes the data <NUM>.

In some embodiments, Prefetcher0 may not perform a prefetch operation unless it first verifies that there is free memory available in memory 318b (DRAM1) at the consumer device. In some embodiments, the prefetcher 324a may be implemented, for example, using combinational and/or sequential logic, one or more neural networks, and/or the like.

At operation (<NUM>), Prefetcher0 may push the prefetched data <NUM> to DRAM1 at the consumer device.

In some embodiments, GPU1 may become aware of the presence of the pushed data using various techniques in accordance with example embodiments of the invention. For example, in embodiments in which the Prefetcher may allocate the memory for the pushed data, GPU1 may check a reserved memory area that may be allocated for the pushed data. As another example, GPU1 may be aware of the presence of the pushed data by checking page table data.

<FIG> illustrates an example embodiment of a host-based memory allocation method in accordance with example embodiments of the invention. The embodiment illustrated in <FIG> may be used, for example, with any of the systems and/or methods disclosed herein, but for purposes of illustration, the embodiment illustrated in <FIG> may be described in the context of the system illustrated in <FIG> which is shown in simplified form in <FIG>.

Referring to <FIG>, at operation (<NUM>) GPUO may write first data to Storage0, which may be observed by Prefetcher0. At operation (<NUM>), GPU1 may read the first data from Storage0, which may also be observed by Prefetcher0. Based on operations (<NUM>) and (<NUM>), Prefetcher0 may detect an access pattern between GPUO and GPU1. Thus, at operation (<NUM>) Prefetcher0 may send a request to host device <NUM> to allocate target memory in DRAM1 for additional data transfers to DRAM1. The request may include, for example, the consumer GI ID for GPU1, the size (amount) of data to transfer, and a logical block address (LBA) indicating the location of the data to transfer.

At operation (<NUM>), the host device <NUM> may allocate the requested memory space in DRAM1. In some embodiments, the CPU <NUM> of host device <NUM> may initiate a direct memory access (DMA) transfer of second data from Storage0 to DRAM1 which may be performed at operation (<NUM>). In other embodiments, Prefetcher0 may initiate and/or perform the data transfer (e.g., by prefetching the data and pushing it to DRAM1) after the host device <NUM> completes the memory allocation.

<FIG> illustrates an example embodiment of a unified memory architecture in accordance with example embodiments of the invention. The embodiment illustrated in <FIG> may be used, for example, to implement the host-based memory allocation method illustrated in <FIG>. For purposes of illustration, the embodiment illustrated in <FIG> may be described in the context of the system illustrated in <FIG>.

Referring to <FIG>, the architecture may implement a shared virtual address space <NUM> having virtual memory addresses (VMAs) such that the CPU <NUM> may be aware of the memory usage in DRAM0, DRAM1, DRAM2, and DRAM3. The memory manager <NUM> (e.g., a VMM) may be located at the host device <NUM> to enable the host device <NUM> to perform the memory allocation. The host device <NUM> may also run an application <NUM> and execute a device kernel driver <NUM>. In some embodiments, the shared virtual address space <NUM> may be used to map, for example, Tier <NUM> (T1) memory, Tier <NUM> (T2) memory, and/or host memory to one or more compute devices <NUM> and/or storage devices <NUM>. In some embodiments, a coherency engine (e.g., a CXL coherency engine at the host device <NUM>) may maintain coherency between the memories illustrated in <FIG>.

<FIG> illustrates an example embodiment of a storage device-based memory allocation method in accordance with example embodiments of the invention. The embodiment illustrated in <FIG> may be used, for example, with any of the systems and/or methods disclosed herein, but for purposes of illustration, the embodiment illustrated in <FIG> may be described in the context of the system illustrated in <FIG> which is shown in simplified form in <FIG>.

Referring to <FIG>, the memories <NUM>, 318a, 318b, 318c, and 318d may include reserved areas <NUM>, 319a, 319b, 319c, and 319d, respectively. At operation (<NUM>) GPUO may write first data to Storage0, which may be observed by Prefetcher0. At operation (<NUM>), GPU1 may read the first data from Storage0, which may also be observed by Prefetcher0. Based on operations (<NUM>) and (<NUM>), Prefetcher0 may detect an access pattern between GPUO and GPU1. Thus, at operation (<NUM>) Prefetcher0 may allocate target memory space in a reserved space 319b of DRAM1 for additional data transfers to DRAM1. Prefetcher0 may allocate the target memory space, for example, using a VMM at the storage device 306a.

Prefetcher0 may then prefetch and copy additional data to the allocated target space in the reserved space 319b of DRAM1. At operation (<NUM>), Prefetcher0 may send a request to the host device <NUM> to update one or more page table mappings of the newly allocated space.

<FIG> illustrates an example embodiment of a memory allocation method in accordance with example embodiments of the invention. The embodiment illustrated in <FIG> may be used, for example with any of the systems and/or methods disclosed herein.

Referring to <FIG>, at operation <NUM>, a prefetcher may initiate a memory allocation operation that may be performed, for example, by a request through a host, or by the prefetcher itself. If the prefetcher decides to have the memory allocation performed by the host, it may proceed to operation <NUM> where the prefetcher may send a memory allocation request to a CPU of a host device. The prefetcher may send the request, for example, to a VMM on a host CPU side of the system. At operation <NUM>, as part of the request, the prefetcher may include information such as the consumer GI ID for the GPU at the consumer device for which the memory is to be allocated, the size (amount) of data to transfer, and an LBA indicating the location of the data to transfer. At operation <NUM>, the VMM at the host device may allocate the requested memory in the device memory at the consumer device corresponding to the GI ID of the GPU. At operation <NUM>, after allocating the target memory space for the consumer device, the host may trigger a DMA transfer of data from the storage device at which the requesting prefetcher is located, and the target memory at the consumer device. The host may also update a page table to reflect the newly allocated target memory at the consumer device.

If, however, the prefetcher decides to allocate the target memory itself, then at operation <NUM>, the prefetcher may initiate the allocation with a VMM at the prefetcher. At operation <NUM>, the VMM may allocate the target memory at the consumer device, for example, from a reserved memory area. At operation <NUM>, the prefetcher may prefetch the data and copy it to the target memory at the consumer device. At operation <NUM>, the prefetcher may request the host device to update a page table to reflect the newly allocated target memory at the consumer device.

<FIG> illustrates an example embodiment of a method for storing, prefetching, and transferring data method in accordance with example embodiments of the invention. The embodiment illustrated in <FIG> may be used, for example, with any of the systems and/or methods disclosed herein, but for purposes of illustration, the embodiment illustrated in <FIG> may be described in the context of the system illustrated in <FIG>. Thus, GPU0, DRAM1, GPU1, CPU, Prefetcher0, and Storage Device may refer to elements 316a, 318b, 316b, <NUM>, 324a, and 306a, respectively, in <FIG>.

Referring to <FIG>, the method may begin at operation <NUM> when the CPU may send one or more indications of producer-consumer relationships to Prefetcher0. At operation <NUM>, Prefetcher0 may store one or more Gl ID and/or information about producer-consumer relationships.

At operation <NUM>, GPU0, at the producer device, may begin writing first data to the Storage Device. At operation <NUM>, a CPU coherency engine may send a producer (e.g., initiator) GI ID for GPUO to Prefetcher0, for example, using one or more cxl. mem fields such as the tag field. At operation <NUM>, Prefetcher0 may determine a stream in which to place the first data from GPUO and store the first data via a multi-stream interface based, for example, on one or more of the stored indications and/or the determined placement. At operation <NUM>, GPUO may notify Prefetcher0 that the write operation of the first data is complete, for example, by writing any data to a predetermined memory location.

At operation <NUM>, GPU1 may begin a read operation of the first data from the Storage Device (which was written by GPUO). At operation <NUM>, the CPU coherency engine may send a consumer GI ID for GPU1 to Prefetcher0, for example, using one or more cxl. mem fields such as the tag field. At operation <NUM>, Prefetcher0 may send the first data from the Storage Device to GPU1. At operation <NUM>, Prefetcher0 may detect a runtime access pattern between GPUO and GPU1 based on the write and read operations <NUM> and <NUM>. In some embodiments, the Prefetcher may not detect this pattern, for example, if the CPU has sent one or more indications of a producer-consumer relationship between GPUO and GPU1.

At operation <NUM>, Prefetcher0 may initiate a memory allocation for target memory at DRAM1 with the VMM. If the Prefetcher initiates a memory allocation by requesting a memory allocation from the host CPU, the VMM located at the host device may perform the allocation. If, however, Prefetcher0 performs the memory allocation itself, it may use the VMM located at the Storage Device. At operation <NUM>, the VMM (whether at the host CPU or Storage Device) may allocate target space in DRAM1. At operation <NUM>, Prefetcher0 may prefetch the data from the stream in which it was stored. At operation <NUM>, Prefetcher0 may push the prefetched data to DRAM1. At operation <NUM>, Prefetcher0 may request the host CPU to update a page table for the data pushed to DRAM1.

<FIG> illustrates an example embodiment of a heterogeneous memory control system in accordance with example embodiments of the invention.

The embodiment illustrated in <FIG> may include an Advanced Configuration and Power Interface (ACPI) Root Table <NUM>, a system resource affinity table (SRAT) <NUM>, and a heterogeneous memory attributes table (HMAT) <NUM>, which may be used to implement Memory Proximity Domain Attributes Structure(s) <NUM>, System Locality Latency and Bandwidth Information Structure(s) <NUM>, and Memory Side Cache Information Structure(s) <NUM>, which in turn may implement one or more Memory Proximity Domains <NUM>, one or more Proximity Domains <NUM>, and/or one or more Proximity Domain Numbers (<NUM>).

The embodiment illustrated in <FIG> may be used, for example, to use one or more CXL features to obtain GI IDs for one or more GPUs at compute devices, prefetchers at storage devices, I/O devices, and/or the like. Additionally, the ACPI Root Table <NUM>, SRAT <NUM>, and or HMAT <NUM> may provide information about processors, memory ranges, GIs, (e.g., heterogeneous processors, accelerators, GPUs, and/or I/O devices with integrated compute or DMA engines). In some implementations, some or all requests from a first CXL device to a second CXL device may be routed through the host. However, in some systems in accordance with example embodiments of the invention, a host CPU may pass producer and/or consumer GI ID information to a prefetcher (e.g., at a storage controller), for example, a cxl. mem tag and/or metavalue + metafield fields.

<FIG> illustrates an example embodiment of a host apparatus that may be used to implement data prefetching and transfer in accordance with example embodiments of the invention. The host apparatus <NUM> illustrated in <FIG> may include a processor <NUM>, which may include a memory controller <NUM>, a system memory <NUM>, a memory allocator <NUM>, a VMM <NUM> and/or a interconnect interface <NUM>, which may be implemented, for example using CXL. Any or all of the components illustrated in <FIG> may communicate through one or more system buses <NUM>. In some embodiments, the host apparatus <NUM> illustrated in <FIG> may be used to implement any of the host functionality disclosed herein including any of the functionality relating to providing one or more indications of producer-consumer relationships to a prefetcher, and/or allocating memory in a compute device for pushed data. In some embodiments, one or more of the components illustrated in <FIG> may be implemented using other components. For example, in some embodiments, one or more of the memory allocator <NUM> and/or VMM <NUM> may be implemented, for example, by the processor <NUM> executing instructions stored in the system memory <NUM> or other memory.

<FIG> illustrates an example embodiment of a device that may be used to implement data prefetching and transfer in accordance with example embodiments of the invention. The device <NUM> includes a prefetcher <NUM> which may include detection logic <NUM>, a storage medium <NUM>, and an interconnect interface <NUM>, and may include a device controller <NUM>, a multi-stream interface <NUM>, a VMM <NUM>, and a media translation layer <NUM>. The components illustrated in <FIG> may communicate through one or more device buses <NUM>. In some embodiments, the device <NUM> illustrated in <FIG> may be used to implement any of the prefetching and/or data pushing functionality disclosed herein.

Any of the functionality described herein, including any of the host functionality, device functionally, and/or the like described with respect to <FIG>, for example, a prefetcher, detection logic, and/or the like, may be implemented with hardware, software, or any combination thereof including combinational logic, sequential logic, one or more timers, counters, registers, state machines, volatile memories such as DRAM and/or static random access memory (SRAM), nonvolatile memory and/or any combination thereof, complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), central processing units (CPUs) such as complex instruction set computer (CISC) processors such as x86 processors and/or reduced instruction set computer (RISC) processors such as ARM processors, graphics processing units (GPUs), neural processing units (NPUs), and/or the like, executing instructions stored in any type of memory. In some embodiments, one or more components may be implemented as a system-on-chip (SOC).

Any of the storage devices disclosed herein may be implemented in any form factor such as <NUM> inch, <NUM> inch, <NUM> inch, M. <NUM>, Enterprise and Data Center SSD Form Factor (EDSFF), NF1, and/or the like, using any connector configuration such as Serial ATA (SATA), Small Computer System Interface (SCSI), Serial Attached SCSI (SAS), U. <NUM>, and/or the like. Any of the storage devices disclosed herein may be implemented entirely or partially with, and/or used in connection with, a server chassis, server rack, dataroom, datacenter, edge datacenter, mobile edge datacenter, and/or any combinations thereof.

<FIG> illustrates aspects of an embodiment of a method for transferring data in accordance with example embodiments of the invention. The method may begin at operation <NUM>. At operation <NUM>, the method writes, by a producer device, data to a storage device through an interconnect. For example, a GPU may write the results of a first computation as first data to the storage device. At operation <NUM>, the method determines a consumer device for the data. For example, a consumer device for the data may form the next stage of a pipeline that may use the first data as an input for a computation at the next stage. At operation <NUM>, the method prefetches the data from the storage device. At operation <NUM>, the method transfers, based on the determining, the data to the consumer device through the interconnect. For example, the prefetcher may push the prefetched data to memory at the consumer device.

Claim 1:
A method for transferring data (<NUM>, <NUM>), the method comprising:
receiving, at a prefetcher (<NUM>, <NUM>, <NUM>) for the storage device (<NUM>, <NUM>, <NUM>, <NUM>), an indication of a relationship between a producer device and a consumer device;
writing (<NUM>), from the producer device, first data (<NUM>) to a storage device (<NUM>, <NUM>, <NUM>, <NUM>) through an interconnect (<NUM>, <NUM>, <NUM>), the first data (<NUM>) including second data (<NUM>); determining (<NUM>) the consumer device for the second data (<NUM>) based on the indication;
prefetching (<NUM>) the second data (<NUM>) from the storage device (<NUM>, <NUM>, <NUM>, <NUM>); and
transferring (<NUM>), based on the determining, the second data (<NUM>) to the consumer device through the interconnect (<NUM>, <NUM>, <NUM>), and
characterized in that
the indication is provided, through the interconnect (<NUM>, <NUM>, <NUM>) to the prefetcher (<NUM>, <NUM>, <NUM>), by an application (<NUM>, <NUM>) associated with the consumer device; and
the second data (<NUM>) is prefetched while the first data (<NUM>) is written.