Patent Description:
A data processing system may use one or more accelerator devices to increase the throughput of the system. Some data processing workloads, such as machine learning workloads, may involve the use of models that may use large amounts of memory. A model may be partitioned and spread across multiple accelerator devices. A portion of the model may be stored at each accelerator device which may perform operations for the corresponding portion of the model.

The above information disclosed in this Background section is only for enhancement of understanding of the background of the invention and therefore it may contain information that does not constitute prior art.

<CIT> discloses a system to compute and distribute data for distributed training of a neural network, the system including first memory to store a first set of instructions including a machine learning framework; a fabric interface to enable transmission and receipt of data associated with the set of trainable machine learning parameters; a first set of general-purpose processor cores to execute the first set of instructions, the first set of instructions to provide a training workflow for computation of gradients for the trainable machine learning parameters and to communicate with a second set of instructions, the second set of instructions facilitate transmission and receipt of the gradients via the fabric interface; and a graphics processor to perform compute operations associated with the training workflow to generate the gradients for the trainable machine learning parameters.

<CIT> discloses: A method, accelerator system, and computer program access data in an out-of-core processing environment. A data access configuration is received from a server system managing a plurality of data sets. A determination is made that data sets retrieved from the server system are to be stored locally based on the data access configuration. A request to interact with a given data set is received from a user client. At least a portion of the given data set is retrieved from the server system. The at least a portion of the given data set is stored locally a memory based on the data access configuration that has been received.

<NPL>) discloses pipelined model parallelism and data parallelism.

<CIT> discloses: An information processing device includes a processing circuit. The processing circuit obtains operation information; estimates, based on the obtained operation information, the execution performance of memory accesses with respect to a first memory and a nonvolatile memory unit in the case in which a managing device performs operations according to each of a plurality of memory control methods; selects, based on the execution performance estimated for each memory control method, any one memory control method from among a plurality of memory control methods; and performs a setting operation with respect to an access managing unit in such a way that the managing device accesses the first memory and the nonvolatile memory unit according to the selected memory control method.

Specific embodiments are defined in the dependent claims.

The figures are not necessarily drawn to scale and elements of similar structures or functions may generally be represented by like reference numerals or portions thereof for illustrative purposes throughout the figures. The figures are only intended to facilitate the description of the various embodiments described herein. The figures do not describe every aspect of the teachings disclosed herein and do not limit the scope of the claims. To prevent the drawings from becoming obscured, not all of the components, connections, and the like may be shown, and not all of the components may have reference numbers. However, patterns of component configurations may be readily apparent from the drawings. The accompanying drawings, together with the specification, illustrate example embodiments of the present disclosure, and, together with the description, serve to explain the principles of the present disclosure.

Some models for applications such as graph processing, machine learning (ML), and/or the like, may be too large to fit in the memory of an accelerator device. Therefore, the model may be partitioned and spread across multiple accelerator devices. However, this may increase data transfers between a host and the accelerator devices because the accelerator devices may swap different portions of the model into their memory as they process inputs using different portions of the model.

Some accelerator devices in accordance with example embodiments of the disclosure may include multi-tier memory systems. For example, a first tier may include high-bandwidth memory that may function as working memory for the accelerator. A second tier may include high-capacity memory that may store some or all of a model. Depending on the implementation details, this may reduce the time, cost, energy, and/or the like, of moving portions of a large model (and/or input data for the model) into the accelerator device.

In some embodiments, the second tier may include a cache to improve the access speed of more frequently used portions of data stored in the high-capacity memory (e.g., more frequently used portions of a model). Depending on the implementation details, this may reduce the latency of the second tier of memory, which in turn, may improve the performance of the accelerator device in applications such as ML inference which may be latency-sensitive.

Some accelerator devices in accordance with example embodiments of the disclosure may implement one or more virtual accelerators. For example, one or more resources of a physical accelerator (e.g., processor cores, working memory, cache, and/or the like) may be partitioned into multiple virtual accelerators, each of which may appear as a separate accelerator to a host or other device. Depending on the implementation details, this may enable an accelerator device to implement model, data, and/or workload parallelism.

Some embodiments may include a memory switch and/or a memory manager. For example, a memory switch may configure physical connections to, from, and/or between various types of memory that may be used for a first memory tier, a second memory tier, cache, and/or the like, whereas a memory manager may manage data movement between the memory devices and/or between the accelerator device and a host and/or other devices. In some embodiments, a memory manager may control data movement based on one or more learned memory access patterns.

In some embodiments, various amounts of control may be provided to a user and/or an application, for example, through one or more registers and/or application programming interfaces (APIs) that may determine the allocation of resources between virtual accelerators, the allocation of memory resources in a multi-tier memory system, and/or the like.

In some embodiments, a workflow and/or a model such as a graph, a machine learning model, a neural network, and/or the like, may be partitioned between multiple accelerator devices and/or virtual accelerators in accordance with example embodiments of the disclosure. For example, a host may partition a model between virtual accelerators based on the memory requirements and/or compute times of the portions of the model, as well as the memory resources and/or cores of the virtual accelerators. In some embodiments, based on the partitioning, the host may generate a clustered graph with data groups to be executed by the virtual accelerators and scheduled by a memory manager.

The principles disclosed herein have independent utility and may be embodied individually, and not every embodiment may utilize every principle. However, the principles may also be embodied in various combinations, some of which may amplify the benefits of the individual principles in a synergistic manner. For example, some accelerator devices may implement multi-tier memory systems without virtual accelerators, and other accelerators may implement virtual accelerators without tiered memory. However, some embodiments may implement a multi-tier memory system and virtual accelerators in the same device which, depending on the implementation details, may amplify the individual benefits of both features.

<FIG> illustrates an embodiment of a framework for analyzing parallel processing operations in accordance with example embodiments of the disclosure. The framework illustrated in <FIG> may be used, for example, to analyze graph processing and/or deep learning (DL) applications (e.g., deep neural networks (DNNs)) in which computations and/or portions of a DL model may be distributed across multiple machines such as accelerator devices (e.g., multiple neural processing units (NPUs)). In the embodiment illustrated in <FIG>, a model <NUM> may be split across multiple machines Machine <NUM> through Machine <NUM> as described below.

Using one or more accelerator devices with a large model may be difficult, for example, because only a small portion of the model, and/or input data for the model, may fit in the working memory of the accelerator device. Thus, using an accelerator device to perform a computation may involve the following operations: (<NUM>) a portion of a trained model may be moved to the working memory of the accelerator; (<NUM>) one or more inputs for the computation (e.g., one or more vectors) may be moved to the working memory of the accelerator device; (<NUM>) the accelerator device may perform a computation using the one or more inputs and the portion of the trained model; and (<NUM>) the accelerator device may store one or more results of the computation in the working memory and/or send the one or more results to a host or other device. Moreover, to perform a computation for a different portion of the model, operations (<NUM>) through (<NUM>) may be repeated.

Referring to <FIG>, various types of parallelism may be used to reduce processing time in accordance with example embodiments of the disclosure. For example, data parallelism may involve splitting data for a workload across multiple machines (e.g., accelerator devices). In some embodiments, data parallelism may reduce processing time (e.g., training and/or inference time). However, depending on the implementation details, data parallelism may be compute bound (e.g., limited by an amount of compute resources), and thus, processing time may increase if a compute limit is reached.

Model parallelism may involve splitting a model across multiple machines (e.g., accelerator devices), each of which may process data for a specific portion of the model. In some embodiments, model parallelism may reduce processing time, however, depending on the implementation details, model parallelism may be memory bound (e.g., limited by an amount of memory resources). For example, a large DL model may cause a system with model parallelism to reach a memory limit.

Workload partitioning may involve splitting a workload (e.g., data and model) across multiple machines (e.g., accelerator devices). In some embodiments, workload partitioning may reduce processing time. For example, with larger batch sizes, processor core utilization and/or accuracy may be improved and/or training times may be reduced. However, depending on the implementation details, workload partitioning may be memory bound and/or compute bound. For example, a large DL model may cause a system with model parallelism to reach a memory limit, in which case, compute resources (e.g., processor cores) may be underutilized.

With any of the types of parallelism described above, memory factors affecting system performance may be based on the width and/or depth of a machine learning model, a data batch size and/or input quality, and/or the like. Moving data may be expensive in terms of time, energy, and/or the like. Adding additional machines may be effective for workloads with large compute requirements, but the machines may be underutilized (and therefore expensive) with memory bound workloads. Moreover, in some embodiments, implementing parallelism in a hyperscaling environment may place an emphasis on inference workloads and/or low latency workloads. For example, some training workloads may be performed offline where relatively high latency may be acceptable. However, online (e.g., real-time) ML workloads may be more sensitive to latency, regardless of batch size. For example, with small batch size inference workloads such as mobile image recognition, users may expect fast results. Similarly, with large batch inference workloads such as language processing, image search, and/or recommendations for ecommerce and/or social media, users may also expect fast results.

<FIG> illustrates an embodiment of a device having tiered memory in accordance with example embodiments of the disclosure. The device <NUM> illustrated in <FIG> may include an interconnect interface <NUM>, a memory system <NUM>, and an accelerator <NUM>. The memory system <NUM> may include a first memory tier <NUM> and a second memory tier <NUM>. The first memory tier <NUM> may be implemented, at least in part, with a first type memory, and the second memory tier <NUM> may be implemented, at least in part, with a second type memory. The memory system <NUM> and accelerator <NUM> may communicate with the interconnect interface <NUM> through an interconnect <NUM>. The accelerator <NUM> may access the first type of memory <NUM> through a first memory interface <NUM>. The accelerator <NUM> may access the second type of memory <NUM> through a second memory interface <NUM>.

In some embodiments, the first type memory in the first memory tier <NUM> may be implemented with one or more types of memory that may provide relatively high bandwidth. Thus, the first memory tier <NUM> may be referred to as bandwidth-enhanced memory. For example, the first type memory may be implemented with high bandwidth memory (HBM) that may include one or more stacks of memory devices, one or more interposers, and one or more point-to-point interconnects. As another example, the first type memory may be implemented with dynamic random access memory (DRAM) such as double data rate (DDRX) DRAM of any generation where X may indicate a generation (e.g., DDR2, DDR3, DDR4, etc.), low-power double data rate (LPDDRX) DRAM, and/or the like. Other examples may include synchronous DRAM (SDRAM), static random access memory (SRAM), and/or the like. In some embodiments, the first memory tier <NUM> may include a combination of different memory types. In some embodiments, the first memory tier <NUM> may be optimized for bandwidth within the memory system <NUM>.

In some embodiments, the second type memory in the second memory tier <NUM> may be implemented with one or more types of memory that may provide relatively high capacity. Thus, the second memory tier <NUM> may be referred to as capacity-enhanced memory. For example, the second type memory may be implemented with nonvolatile memory which may include flash memory such as not-AND (NAND) flash memory, low-latency NAND flash memory, persistent memory (PMEM) such as cross-gridded nonvolatile memory, memory with bulk resistance change, phase change memory (PCM), and/or the like, or any combination thereof. In some embodiments, the second memory tier may be optimized for capacity within the memory system <NUM>.

In some embodiments, the memory system <NUM> may include a cache <NUM> arranged to cache data for the first memory tier <NUM> and/or the second memory tier <NUM>. The cache <NUM> may be implemented, for example, with relatively low latency memory such as DRAM, SRAM, and/or the like. In some embodiments, the cache <NUM> may be part of the second memory tier <NUM>. Thus, the second memory tier <NUM> may be referred to as latency-enhanced memory. In some embodiments, the second memory tier may be optimized for latency within the memory system <NUM>.

The device <NUM> illustrated in <FIG> may be used to implement a wide variety of applications. For example, in some embodiments, the device <NUM> may be used for ML training and/or inference (e.g., DL and/or DNNs), speech recognition, language processing, image recognition, graph processing, generating recommendations, and/or the like. In some embodiments, the first memory tier <NUM> may be configured as a high-bandwidth memory that may function as working memory for the accelerator, and the second tier may <NUM> be configured as a low-latency, high-capacity memory (e.g., using a cache <NUM>) that may store some or all of a model such as a graph or ML model. Depending on the implementation details, this may reduce the time, cost, energy, and/or the like, of moving portions of a model (and/or input and/or output data for the model) into the accelerator device, improve the access speed of one or more portions of the first and/or second memory tiers, reduce the latency of one or more accelerator operations, and/or the like.

The interconnect interface <NUM> and/or interconnect <NUM> may be implemented, for example, with one or more of any type of interface and/or protocol including Peripheral Component Interconnect Express (PCIe), Nonvolatile Memory Express (NVMe), NVMe-over-fabric (NVMe-oF), Advanced eXtensible Interface (AXI), Ultra Path Interconnect (UPI), Ethernet, Transmission Control Protocol/Internet Protocol (TCP/IP), remote direct memory access (RDMA), RDMA over Converged Ethernet (ROCE), FibreChannel, InfiniBand, Serial ATA (SATA), Small Computer Systems Interface (SCSI), Serial Attached SCSI (SAS), iWARP, and/or the like, or any combination thereof. In some embodiments, the interconnect interface <NUM> may be implemented with one or more memory semantic and/or memory coherent interfaces and/or protocols such as Compute Express Link (CXL), and/or CXL. io, and/or CXL. cache, Gen-Z, Coherent Accelerator Processor Interface (CAPI), Cache Coherent Interconnect for Accelerators (CCIX), and/or the like, or any combination thereof.

The memory interfaces <NUM> and <NUM> may be implemented, for example, with one or more of any type of interface including DDRX, LPDDRX, Open Memory Interface (OMI), NVLink, High Bandwidth Memory (HBM), HBM2, HBM3, and/or any of the interconnect interfaces and/or protocols mentioned above including CXL. The memory interfaces <NUM> and <NUM> may be implemented with coherent and/or non-coherent interfaces. For example, a non-coherent memory interface may be used for the memory interface <NUM> between the first memory tier <NUM> and the accelerator <NUM>, while a coherent interface may be used for the memory interface <NUM> between the second memory tier <NUM> and the accelerator <NUM>.

One or more of the interconnect interface <NUM>, interconnect <NUM>, and memory interfaces <NUM> and <NUM> may be implemented as separate components or integrated into an interconnect fabric, for example, using one or more switches to configure connections between the components illustrated in <FIG>.

The accelerator <NUM> may be implemented with any type of device that may include one or more processing resources suitable for an accelerator, for example, a graphics processing unit (GPU), a neural processing unit (NPU), tensor processing unit (TPU), an accelerator based on combinational logic, sequential logic, one or more timers, counters, registers, state machines, complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), central processing units (CPUs) such as complex instruction set computer (CISC) processors such as x86 processors and/or reduced instruction set computer (RISC) processors such as ARM processors and/or the like, or any combination thereof.

The device <NUM> may be implemented in any physical and/or electrical configuration and/or form factor such as a free-standing apparatus, an add-in card such as a PCIe adapter or expansion card, a plug-in device, for example, that may plug into a connector and/or slot of a server chassis (e.g., a connector on a backplane and/or a midplane of a server or other apparatus), and/or the like. In some embodiments, the device <NUM> may be implemented in a form factor for a storage device such as <NUM> inch, <NUM> inch, <NUM> inch, M. <NUM>, Enterprise and Data Center SSD Form Factor (EDSFF), NF1, and/or the like, using any connector configuration for the interconnect interface <NUM> such as a SATA connector, SCSI connector, SAS connector, M. <NUM> connector, U. <NUM> connector, U. <NUM> connector, and/or the like. Any of the devices disclosed herein may be implemented entirely or partially with, and/or used in connection with, a server chassis, server rack, dataroom, datacenter, edge datacenter, mobile edge datacenter, and/or any combinations thereof. In some embodiments, the device <NUM> may be implemented as a CXL Type-<NUM> device, a CXL Type-<NUM> device, a CXL Type-<NUM> device, and/or the like.

<FIG> illustrates an embodiment of a device with accelerator virtualization in accordance with example embodiments of the disclosure. The device <NUM> illustrated in <FIG> may include an interconnect interface <NUM>, a memory system <NUM>, an accelerator <NUM>, and an interconnect <NUM> that may be similar to those described above with respect to <FIG>, although the memory system <NUM> may or may not implement a tiered memory structure and may instead use any type of memory or combination thereof in any configuration. The accelerator <NUM> may be interfaced to the memory system <NUM> through one or more memory interfaces <NUM> of any type.

The device <NUM> may also include virtualization logic <NUM> that may partition one or more resources of the accelerator <NUM> into one or more virtual accelerators <NUM>-<NUM>,. Accelerator resources that may be partitioned may include processing resources (e.g., processor cores), registers, memory resources, interconnect resources, and/or the like. For example, in some embodiments, the virtualization logic <NUM> may allocate one or more physical processor cores of the accelerator <NUM> to one or more of the virtual accelerators <NUM>-<NUM>,. The virtualization logic <NUM> may also allocate a portion of memory from the memory system <NUM> to one or more of the virtual accelerators <NUM>-<NUM>,. In embodiments in which the memory system <NUM> is implemented with tiers, the virtualization logic may allocate a portion of memory from each tier to one or more of the virtual accelerators <NUM>-<NUM>,. In embodiments in which a memory tier may include a cache, a portion of the cache may be allocated to one or more of the virtual accelerators <NUM>-<NUM>,.

In some embodiments, the virtualization logic <NUM> may also allocate separate interconnect resources such as ports to one or more of the virtual accelerators <NUM>-<NUM>,.

Depending on the implementation details, one or more of the virtual accelerators <NUM>-<NUM>,. , <NUM>-N may appear as separate accelerators to a host or other device. For example, each of the virtual accelerators <NUM>-<NUM>,. , <NUM>-N may be implemented as a separate endpoint having a separate device identifier (ID). In some embodiments, virtual accelerators may be implemented as separate endpoints by implementing the device <NUM> as a multi-headed device in which each of the virtual accelerators <NUM>-<NUM>,. , <NUM>-N may have a separate port.

The virtual accelerators <NUM>-<NUM>,. , <NUM>-N may be implemented with various amounts of isolation between the virtual accelerators. For example, in some embodiments, all of the resources (e.g., processor cores, memory, ports, and/or the like) for each virtual accelerator may be completely isolated such that the individual virtual accelerators may not compete for resources, and/or each virtual accelerator may not pollute the memory (e.g., cache) of any other virtual accelerator. As another example, in some embodiments, one or more resources (e.g., an interconnect port, a portion of cache, and/or the like) may be partially or fully shared between virtual accelerators.

In some embodiments, the virtualization logic <NUM> may be programmable, for example, using one or more registers <NUM> that may be accessed by a user or application through the interconnect interface <NUM> (e.g., using an API). For example, the virtualization logic <NUM> may be programmed to configure a first virtual accelerator <NUM>-<NUM> to perform a first operation on a first portion of data received through the interconnect interface <NUM> and stored in the memory system <NUM>. The virtualization logic <NUM> may further be programmed to configure a second virtual accelerator <NUM>-<NUM> to perform a second operation on a second portion of the data received through the interconnect interface <NUM> and stored in the memory system <NUM>. In some embodiments, the first and second portions of the data may be stored in separate portions of the memory system <NUM> that may be allocated to the separate virtual accelerators <NUM>-<NUM> and <NUM>-<NUM>.

A device with accelerator virtualization in accordance with example embodiments of the disclosure may be used in a wide variety of applications. For example, in some embodiments, a model (e.g., a graph, an ML model, and/or the like) may be partitioned into portions that may each be assigned to a virtual accelerator to implement model parallelism. Depending on the implementation details, this may enable a relatively large model to be implemented efficiently across multiple virtual accelerators. Moreover, the use of virtual accelerators in accordance with example embodiments of the disclosure may reduce or eliminate memory bound limitations, especially if the virtual accelerators are implemented in conjunction with a tiered memory system in accordance with example embodiments of the disclosure.

For purposes of illustrating the principles of this disclosure, some example embodiments are described below in the context of systems, methods, and/or devices that may use specific implementation details such as CXL interconnects and/or protocols, CXL Type-<NUM> devices, DRAM for caches, flash memory for capacity-enhanced memory, NPUs for accelerators, DL and/or graph models, and/or other implementation details. The principles, however, are not limited to these example implementation details and may be applied to embodiments that may use any other interconnects, protocols, memory types, device types, accelerators, models, and/or the like.

<FIG> illustrates an example embodiment of a host in accordance with example embodiments of the disclosure. <FIG> illustrates an example embodiment of a device with tiered memory and virtual accelerators in accordance with example embodiments of the disclosure. <FIG> and <FIG> are collectively illustrate an example embodiment of a system in accordance with example embodiments of the disclosure and are referred to collectively as <FIG>. The embodiment illustrated in <FIG> may be used, for example, to implement any of the embodiments described above (including one or more of the features thereof) with respect to <FIG>.

Referring to <FIG>, the system may include a device <NUM> and a host <NUM> that may communicate through an interconnect <NUM>. In some embodiments, the system may further include remote storage <NUM> that may communicate with the host <NUM> through a network <NUM> and network interface <NUM> as shown in <FIG>, but in other embodiments, the remote storage <NUM> may be connected to the system in any other manner, for example, through one or more interconnects such as interconnect <NUM> using a switch.

The device <NUM> may include a tiered memory system <NUM> having a bandwidth-enhanced memory <NUM> and a latency-enhanced memory <NUM> that may include capacity-enhanced memory <NUM> and a cache <NUM>. The bandwidth-enhanced memory <NUM> may be implemented, at least partially, with one or more bandwidth-enhanced point-to-point memory devices such as LPDDRX devices and/or HBM devices that may include a stack of memory dies with an interposer. The latency-enhanced memory <NUM> may be implemented, at least partially, with capacity-enhanced memory <NUM> such as low-latency NAND memory and a DRAM cache <NUM>.

The device <NUM> may also include one or more accelerators <NUM> which, in this example, may be implemented as an NPU. The NPU <NUM> may be partitioned into one or more virtual NPUs <NUM>-<NUM>,. , <NUM>-N by virtualization logic <NUM> which may be programmed, for example, using one or more registers <NUM>. The virtualization logic <NUM> may operate, for example, in a manner similar to the virtualization logic <NUM> described above with respect to <FIG>.

Referring to <FIG>, the virtual NPUs <NUM>-<NUM>,. , <NUM>-N may be interfaced to the bandwidth-enhanced memory <NUM> through a non-coherent memory interface <NUM> such as DDRX and/or NVLINK. The virtual NPUs <NUM>-<NUM>,. , <NUM>-N may be interfaced to the latency-enhanced memory <NUM> through a coherent interface <NUM> such as CXL. Although the memory interfaces <NUM> and <NUM> are shown as separate interfaces, in some embodiments, they may be integrated, for example, as a memory interface fabric. In some embodiments, some or all of the bandwidth-enhanced memory <NUM> may function as a working memory for the NPU <NUM> and/or one or more of the virtual NPUs <NUM>-<NUM>,.

The device <NUM> may also include an interconnect interface <NUM> which, in this example, may be implemented as a CXL interface and use a CXL. io protocol interface <NUM>, a CXL. cache protocol interface <NUM>, and/or a CXL. mem protocol interface <NUM>. A device coherency (DCOH) engine <NUM> (e.g., a coherency agent) may resolve coherency of one or more caches at the device and/or manage one or more CXL bias states.

As claimed, the device <NUM> also includes a memory switch <NUM>, which configures physical connections to, from, and/or between various types of memory that may be used for first tier memory, second tier memory, cache, and/or the like. Although the memory switch <NUM> is shown as a single component, in some embodiments, it may be implemented in a distributed manner with one more portions of the memory switch <NUM> located within and/or between any of the components of the tiered memory system <NUM>, any components of the NPU <NUM>, and/or the like, for example, as shown by the overlap between the memory switch <NUM>, the tiered memory system <NUM>, and the NPU <NUM> in <FIG>. Thus, in some embodiments, the memory switch <NUM>, and/or any components of the tiered memory system <NUM> and/or NPU <NUM> may form a memory fabric in which the memory switch <NUM> may function as a memory fabric manager. In some embodiments, the memory manager may implement one or more memory configurations using topologies such as torus, mech, point-to-points, and/or the like.

As claimed, the memory switch <NUM> Z is programmable, for example, by an application through an API, to configure the memory switch <NUM>, and/or any components of the tiered memory system <NUM> and/or NPU <NUM>. For example, a programmer may know that a certain workload to run on the NPU <NUM> or one of the virtual NPUs <NUM>-<NUM>,. , <NUM>-N may require, or benefit from, a specific amount of cache <NUM>. Thus, the programmer (e.g., through an application) programs the memory switch (e.g., through an API) to configure the specific amount of cache to the NPU <NUM> or one of the virtual NPUs <NUM>-<NUM>,.

In some embodiments, the device <NUM> may also include a memory manager <NUM>. The memory manager <NUM> may manage data movement within the tiered memory system <NUM>, between any components of the tiered memory system <NUM> and the NPU <NUM>, between any components of the tiered memory system <NUM> and a host and/or other devices, and/or the like. Thus, the memory switch <NUM> may configure a physical arrangement of memory and other resources, whereas the memory manager <NUM> may manage data movement within the configured memory arrangement. In some embodiments, the memory switch <NUM> and or the memory manager <NUM> may configure memory and/or control data movement based on one or more observed or learned memory access patterns as determined, for example, by a memory access pattern analyzer <NUM>. In some embodiments, the memory manager <NUM> may implement one or more cache algorithms and/or cache policies.

In some embodiments, the device <NUM> may also include one or more preprocessors <NUM>, one or more input and/or output (I/O or IO) engines <NUM>, and/or one or more compression and/or decompression logic <NUM>. A preprocessor may perform any function that may prepare data for processing by the NPU <NUM> and/or one or more virtual NPUs <NUM>-<NUM>,. For example, preprocessing may be used for data cleaning (e.g., eliminating noise, filling in missing or null values, and/or the like), data transformation (e.g., standardizing, normalization, feature selection, and/or the like), data organization, data reduction (e.g., aggregation, numerosity reduction, dimensionality reduction and/or the like), and/or the like.

An IO engine <NUM> may implement one or more IO related functions such as data deduplication, offloading preprocessing of all or a portion of a dataset (e.g., to a preprocessor <NUM>), and/or the like. The compression and/or decompression logic <NUM> may help improve the effective bandwidth of the interconnect interface <NUM> by compressing data before sending it out, and/or decompressing data after receiving it, through the interconnect interface <NUM>.

In some embodiments, the device <NUM> may also include an allreduce engine <NUM> that may perform one or more operations associated with an allreduce operation. For example, the allreduce engine <NUM> may help schedule transfers of data (e.g., tensors, updates from NPUs, and/or the like) between NPUs, devices, and/or the like.

Some embodiments may include one or more internal interconnects between any of the various components, some examples of which are identified as 456a and 456b (which may be referred to collectively as <NUM>) in <FIG>. Any of the internal interconnects <NUM> may be implemented with any type of bus, point-to-point connection, and/or the like. In some embodiments, one or more internal interconnects <NUM> may be implemented with PCIe, NVLink, AXI, and/or the like. In some embodiments, one or more of the internal interconnects <NUM> may be implemented with packet processing (e.g., to increase the bandwidth of data movement within and/or between any of the components illustrated in <FIG>). Packet processing may be implemented, for example, using one or more network on chip (NOC) devices to perform packet processing on any of the internal interconnects <NUM>.

The host <NUM> may include a CPU <NUM> having an interface (e.g., CXL) controller <NUM>, a memory controller <NUM>, and an internal cache <NUM>. The memory controller <NUM> may control the cache <NUM> and/or one or more host (e.g., system) memories <NUM>. In some embodiments, the cache may be implemented with SRAM, and the host memory <NUM> may be implemented with DRAM.

The host <NUM> may include an interconnect interface <NUM> which, in this example, may be implemented as a CXL interface and use one or more of the CXL. cache, and/or CXL. mem protocols.

The host <NUM> may also include local storage <NUM> which may be implemented, for example, with any type of storage device(s) based on any type of memory and/or storage media including solid state media, magnetic media, optical media, and/or the like.

The host <NUM> may also include a network interface <NUM> that may provide access to remote storage <NUM> and/or any other systems, hosts, devices, and/or the like. The network interface <NUM> may be implemented, for example, as a network interface card (NIC) that may use any suitable networking interface and/or protocol including Ethernet, TCP/IP, RDMA, ROCE, and/or any other interfaces and/or protocols including those mentioned above. The remote storage <NUM> may be implemented, for example, with any type of storage device(s) based on any type of memory and/or storage media including solid state media, magnetic media, optical media, and/or the like, and configured in a server chassis, server rack, dataroom, datacenter, edge datacenter, mobile edge datacenter, and/or any combinations thereof.

The CPU <NUM>, local storage <NUM>, and/or network interface <NUM> may communicate, for example, through a system bus <NUM>.

The host <NUM> may also include accelerator (e.g., NPU) virtualization logic <NUM> which may work in cooperation with the virtualization logic <NUM> at the device <NUM> to partition, manage, and/or use one or more virtual NPUs <NUM>-<NUM>,. Either of the host-side virtualization logic <NUM> and/or device-side virtualization logic <NUM> may be implemented in hardware, software, or a combination thereof. For example, in some embodiments, the device-side virtualization logic <NUM> may be implemented primarily in hardware to partition resources associated with the NPU <NUM> into one or more virtual NPUs <NUM>-<NUM>,. , <NUM>-N, whereas the host-side virtualization logic <NUM> may be implemented primarily in software, for example, at least partially in a device driver for the device <NUM> and/or at least partially in one or more applications <NUM>. In some embodiments, a device driver in host-side virtualization logic <NUM> may receive one or more commands from an application <NUM> through an API to program one or more registers <NUM> in the device-side virtualization logic <NUM> to create, manage, and/or use one or more of the virtual NPUs <NUM>-<NUM>,.

In some embodiments, the device-side virtualization logic <NUM> and/or host-side virtualization logic <NUM> (which may be referred to collectively as virtualization logic) may provide flow control for one or more of the virtual NPUs <NUM>-<NUM>,. In some embodiments, flow control may determine that each virtual NPU may saturate (e.g. be limited to) at a certain amount of bandwidth between the virtual NPU and the host <NUM>. In some embodiments, flow control may be provided on a per-partition (e.g., per virtual NPU) basis. This may be referred to as spatial multiplexing which may implement a form of parallelism across multiple virtual NPUs.

In some embodiments, the virtualization logic may include a hardware and/or software scheduler <NUM> to identify command streams that may be multiplexed among virtual NPUs. This may be referred to as time multiplexing and/or scheduling.

In some embodiments, time multiplexing may be combined with spatial multiplexing. For example, ten different command streams may be time multiplexed across ten different virtual NPUs to present the appearance of <NUM> different NPUs. Depending on the implementation details, the time and/or spatial multiplexing may be distributed among virtual NPUs based, for example, on the amount of resources each virtual NPU may have.

In some embodiments, the network interface <NUM> and/or remote storage <NUM> may initiate a DMA transaction to the tiered memory system <NUM>. For such a transaction, the data path may be through the network interface <NUM>, the CXL controller <NUM>, the host-side interconnect interface <NUM>, the interconnect <NUM> (using the CXL. io) protocol, the device-side interconnect interface <NUM>, and the memory switch <NUM>.

In processing the DMA transaction, the memory switch <NUM> may use one or more queues that may be separate from one or more queues used for CXL transactions (e.g., for a vector transfer using a flit). The use of separate queues may avoid conflicts that may be caused by receiving a request from the host via the CXL. mem and/or CXL. cache interfaces while performing a bulk transfer (e.g., DMA) through CXL. Some embodiment may implement a system of priorities for DMA and/or CXL transfers.

In some embodiments, a list of physical address ranges may be maintained, for example, by a remote server, to help initiate peer RDMA requests via the network interface <NUM> using one or more DMA engines. Address ranges may be maintained for one or more devices such as device <NUM> and/or any virtual NPUs or other accelerators within each device.

To perform a DMA transfer to the NPU <NUM>, the initiating device may use a destination (e.g., target) buffer in the device <NUM>. Thus, in some embodiments, the memory manager <NUM> may allocate one or more target buffer locations for DMA transfers.

In some embodiments, the CXL interface <NUM> may implement the CXL. mem, and/or CXL. cache capabilities using host-managed device memory (HDM) which may be allocated from one or more portions of the tiered memory system <NUM>. For example, data transferred by the host <NUM> to the device <NUM> using CXL may be placed in an HDM region of memory. In some embodiments, the host <NUM> may not be aware of any other memory at the device <NUM> other than the HDM memory. One or more memory regions exposed to the host <NUM> may be indicated, for example, by one or more start and/or end addresses in base address registers (BARs). For example, in a device <NUM> having a total of 1TB of memory in second memory tier <NUM>, only 10GB of the 1TB may be exposed to the host <NUM> as HDM.

Any or all of the components and/or capabilities of the device <NUM> may be configured, managed, operated, implemented, and/or the like, with the assistance of firmware that may run, for example, on a RISC processor core on the device <NUM>. For example, in some embodiments, firmware at the device <NUM> may determine how much and/or which type of memory of the tiered memory system <NUM> to expose to the host <NUM> (e.g., as HDM) and/or which device capabilities to expose to the host <NUM>. These determinations may be based, for example, on the total resources available at the device <NUM>, resources that may be committed to other uses, and/or the like.

In embodiments in which the device <NUM> is implemented as a CXL Type-<NUM> device, I/O transactions may follow any generation of PCle protocol using Address Translation Service (ATS). For example, for the device <NUM> to request access to the host memory <NUM> (e.g., using CXL. cache), the device <NUM> may use ATS to make the request because the device <NUM> may not know the actual physical address in the host memory <NUM> for which to make the request. ATS may be implemented, for example, in the CXL controller <NUM> using an input-output memory management unit (IOMMU) and/or a device-side translation lookaside buffer (DTLB) which may cache one or more address translations. In some embodiments, this may enable the device <NUM> to pull data from the host memory <NUM> (e.g., using CXL. cache) and/or remote storage <NUM> on an on-demand basis.

The host <NUM> may also include a memory access pattern analyzer <NUM> that may monitor and/or analyze memory accesses throughout the system including within and between the tiered memory system <NUM>, the NPU <NUM>, and/or one or more of the virtual NPUs <NUM>-<NUM>,. , <NUM>-N to determine one or more memory access patterns that may be used, for example, to configure one or more components of the tiered memory system <NUM>, the NPU <NUM>, and/or the like for more efficient operation. In some embodiments, the memory access pattern analyzer <NUM> may use one or more ML techniques to identify one or more memory access patterns.

The host <NUM> may run any type of applications <NUM> relating to the use of one or more NPUs and/or other types of accelerators that may be implemented by the device <NUM>. For example, the host <NUM> may run one or more applications that may use the device <NUM> to implement graph processing, ML frameworks, including one or more DL frameworks such as TensorFlow, PyTorch, and/or the like.

As mentioned above, using one or more accelerator devices with a large model may be difficult, for example, because only a small portion of the model, and/or input data for the model, may fit in the working memory of the accelerator device. Depending on the implementation details, adding enough additional working memory to hold the entire model may be prohibitively expensive, for example, in terms of economic cost, space, power consumption, and/or the like. Therefore, using one or more accelerator devices with a large model may involve expensive data transfers to and/or from a host to swap different portions of a model into the working memory of an accelerator device.

However, in some embodiments, the system illustrated in <FIG> may be used to implement a large DL model (e.g., a large DNN model), for example, by using capacity-enhanced memory <NUM> such as flash memory (e.g., low-latency NAND memory) in the second memory tier <NUM> which may, depending on the implementation details, be used to hold an entire DL model at an acceptable cost in terms of economics, space, power consumption, and/or the like. Therefore, the system may reduce or eliminate data transfers associated with using an accelerator device with a DL model.

Moreover, the use of cache <NUM> (e.g., DRAM) in the latency-enhanced memory <NUM> may compensate for potential latency issues with the flash memory (e.g., data may be moved quickly from the second memory tier <NUM> to the first memory tier <NUM>).

Moreover, the system may enable the use of multiple techniques for implementing a DL model in accordance with example embodiments of the disclosure. For example, CXL may be used in conjunction with PCle, and therefore, the system and/or the device <NUM> may be able to chose between PCle and CXL transfers depending on the specific situation. For example, direct memory access (DMA) over PCle may be used at a 4KB granularity to transfer a large trained model into the second memory tier (latency-enhanced and/or capacity-enhance memory) <NUM>, e.g., before the trained model for inference. This may be referred to as placing the model behind the NPU <NUM>. In some embodiments, PCIe transfers may be implemented using the CXL. io protocol, which, depending on the implementation details, may operate essentially as PCle.

Runtime input data for the model may be received at the host <NUM>, for example, through the network interface <NUM>. The input data may be transferred from the host <NUM> to the working memory of the NPU <NUM> (e.g., first memory tier <NUM> of bandwidth-enhanced memory) so the NPU <NUM> may process the input data using the model. Although PCle may not provide coherency, it may still be used to transfer input data (e.g., DL vectors) into the working memory for example, by pushing the input data into the working memory and then notifying the NPU <NUM> that the data is available. (For simplicity, in some embodiments, the NPU <NUM> and/or any of the virtual NPUs <NUM>-<NUM>,. , <NUM>-N may be referred to collectively as the NPU <NUM>.

As another example, CXL may be used to transfer input data from the host <NUM> to the working memory <NUM>. In some embodiments, a potential advantage of using CXL to transfer input data (e.g., DL vectors) is that the coherency of CXL may allow the data to be transferred in a passive manner, e.g., on demand by the NPU <NUM> (and/or any of the virtual NPUs <NUM>-<NUM>,. , <NUM>-N). For example, because the NPU <NUM>, the working memory <NUM>, and/or the host <NUM> may be in the same coherency domain, and thus, when the NPU <NUM> may use a cache line to work on, it may make a cache line request. Moreover, CXL may provide for smaller granularity data transfers (e.g., 64B flits). Depending on the implementation details, this may reduce or eliminate unnecessary data transfers and/or enable more efficient data transfers.

In some embodiments, the cache memory <NUM> may be implemented with hardware control, software control, or a combination thereof. For example, in some embodiments, control of some or all of the cache <NUM> may be determined by hardware microarchitecture with little or no software control. Hardware control of the cache <NUM> may be beneficial, for example, where the data has good spatial and/or temporal locality, in which case, offloading the control work to hardware may be more efficient. Hardware control of the cache <NUM> may also be beneficial with small granularity data transfers because it may be difficult for software to migrate small portions of data because of software overhead associated with controlling the cache <NUM>. With hardware based caching, the cache <NUM> may be transparent to the NPU <NUM>, e.g., the NPU <NUM> may only see the capacity-enhanced memory <NUM> (e.g., NAND flash), but with the benefit of reduced latency provided by faster memory (e.g., DRAM) of the cache <NUM>.

As another example, in some embodiments, the cache <NUM> may be controlled primarily through software. For example, software control of the cache <NUM> may enable a programmer (e.g., through an application) to control which input data vectors to prefetch into cache, which portion of a model store in NAND flash to prefetch into cache, and/or the like. Software control of the cache <NUM> may be beneficial, for example, where the data lacks spatial and/or temporal locality (e.g., with streaming data accesses) in which case, a programmer may be able to make better decisions regarding which data to cache. Also, with large granularity data transfers, the software overhead may be smaller and therefore, it may be beneficial to provide the programmer and/or application with greater control of where to place and/or move data.

In some embodiments, the memory switch <NUM> and/or memory manager <NUM> may implement hardware intelligence to identify hot data (e.g., frequently used or likely to be used blocks, pages, lines, and/or the like) to move and/or keep in the cache <NUM> and/or cold data to move out of the cache <NUM>. In some embodiments, the memory switch <NUM> and/or the memory manager <NUM> may implement the control logic for hardware and/or software control of the cache <NUM>. In some embodiments, the memory switch <NUM> and/or memory manager <NUM> may implement stream detection and/or prefetch to facilitate predicting which data to prefetch into cache. In some embodiments, the cache <NUM> may operate as a staging area for portions of a model and/or input data that may be prefetched (e.g., hot vectors and/or hot indices) to reduce or prevent accesses of the capacity-enhanced memory <NUM> (e.g. flash memory).

An NPU <NUM> or other type of accelerator may be implemented, in some embodiments, with a single instruction, multiple data (SIMD) engine and/or multiple streaming units. One or more NPUs <NUM> may be used to implement a dataflow processing engine distributed across one or more clusters, wherein each cluster may include a dataflow engine or SIMD engine, one or more registers, a floating point (FP) unit, and/or a tile of SRAM cache. In some embodiments, a translation lookaside buffer (TLB) may be used to speed up addressing. In some embodiments, one or more multi-tier caches may be implemented across computer clusters.

The host <NUM> and NPU <NUM> may share one or more cache lines, in some embodiments, in a coherent memory space that may be maintained in a coherent state by the CXL controller <NUM> and/or DCOH engine <NUM>. The DCOH engine <NUM> may implement cache line snooping to track the existence and/or state of one or more cache lines on the device side to enable the device <NUM> to respond when the host <NUM> requests a cache line from the device <NUM>. In some embodiments, the DCOH engine <NUM> may respond to a host snoop on a device-to-host (D2H) response channel, for example, because the device may hold dirty data. To implement the CXL. cache interface, the DCOH engine <NUM> may implement response and/or request logic to generate a response to a request from the host <NUM> for a D2H transfer, and/or to generate a request by the device <NUM> for a host-to-device (H2D) transfer. In some embodiments, the DCOH engine <NUM> may enable the device <NUM> to request one or more cache lines from elsewhere in the system besides the host. For example, the device <NUM> may request one or more cache lines from a different device, accelerator, memory extension, memory buffer, and/or the like, and CXL controller <NUM> at the host <NUM> make facilitate a transfer of the requested cache line to the device <NUM>.

The DCOH engine <NUM> may track a bias state such as a device bias and/or a host bias in embodiments in which the device <NUM> is implemented as a CXL Type-<NUM> device. In a device bias state, an NPU or other type of accelerator may access device local memory without a performance penalty that may be associated with a host bias state, however, a host may still access the device local memory using coherent semantics.

Some embodiments of a device <NUM> may provide one or more enhancements to a CXL or other memory coherent and/or memory semantic interface and/or protocol. For example, a coherent region of device memory (e.g., any memory in the tiered memory system <NUM>) may be written by multiple sources such as the host <NUM>, the device <NUM>, and/or other hosts, devices, and/or the like. An implementation of a device <NUM> in accordance with example embodiments of the disclosure may handle writes to the coherent region from these multiple sources by (<NUM>) limiting the coherent region visible to the host (e.g., the HDM), and/or (<NUM>) using the DCOH to order (and/or enforce the order) of the writes. Depending on the implementation details, this may improve performance, for example, where the order of multiple writes from multiple sources may be undefined. Thus, the DCOH may define the order of writes (e.g., strictly, loosely, and/or the like). Similar techniques may be applied to reads.

The host <NUM> may perform read and/or write accesses of a coherent region of the device memory, in some embodiments, using a master to subordinate protocol. Upon a read request, a device may respond with a data and/or no-data response (NDR) field.

The memory manager <NUM> may manage how memory lines may be distributed between the first memory tier <NUM> and the second memory tier <NUM>. The memory manager <NUM> may also implement cache management between the cache (e.g., DRAM) <NUM> and a capacity-enhanced memory <NUM> (e.g., flash memory) in the second memory tier <NUM>. In some embodiments, the memory manager <NUM> may implement hardware based caching and/or software based caching, for example, with one or more parameters to expose both regions of cache <NUM> and capacity-enhanced memory <NUM> to one or more software layers such as an application <NUM> that may implement a framework.

In some embodiments, the memory manager <NUM> may implement a tag cache manager and/or a controller for the cache <NUM> (e.g., a DDR controller if DDR DRAM is used for the cache <NUM>).

In some embodiments, large granularity transfers, e.g., a system-wide copy of an entire model from a network buffer or external storage device to the capacity-enhanced memory <NUM>, may use DMA flows on PCle through the CXL. io protocol interface <NUM>. Smaller granularity transfers may use CXL flows through the CXL. mem protocol interface <NUM>. In some embodiments, in addition to initially populating the capacity-enhanced memory <NUM> with a model, the memory manager <NUM> may enable an application <NUM> on the host <NUM> to access the capacity-enhanced memory <NUM>, either directly or indirectly through a window of cache <NUM>, to facilitate swapping in and/or out one or more portions of a model between the capacity-enhanced memory <NUM> and the cache <NUM>. In some embodiments, the memory manager <NUM> may separate one or more regions of the first memory tier <NUM> and the second memory tier <NUM> that may be exposed to the host <NUM> in a manner that may be configurable, for example, through one or more registers in the memory manager <NUM> that may be accessed through an API. Thus, in some embodiments, a large DL model may be swapped into and/or out of the capacity-enhanced memory <NUM> (e.g., NAND flash) for initial population before runtime. However, during runtime such as using the model for inference, there may be one or more portions of the model that may be accessed more frequently, which may be facilitated by caching the one or more portions of the model using the cache <NUM> between the tiers <NUM> and <NUM>.

A device memory manager <NUM> may be included in some embodiments to manage one or more aspects of memory devices, storage devices, and/or the like within the tiers <NUM> and <NUM>. Whereas the memory manager <NUM> may implement cache algorithms, manage data movement between tiers <NUM> and <NUM>, and/or the like, the device memory manager <NUM> may perform lower level functions such as wear leveling, address translation, hash tables, tag management, extra logic, data movement within memory devices, storage devices, and/or the like. In some embodiments, the device memory manager <NUM> may implement a signaling scheme that may enable an application <NUM> (e.g., a DL framework) to indicate one or more base addresses, offset addresses, and/or the like for active portions of a model. This may enable the device memory manager <NUM> to move data transparently to the host and/or NPU using hardware. In some embodiments, the device memory manager <NUM> may implement one or more hardware techniques to determine bandwidth sensitive portions of memory without involvement by the host <NUM>, NPU <NUM>, and/or the like. In some embodiments, the device memory manager <NUM> may expose one or more controls through an API, for example, to enable a programmer who may know a wear level pattern for a specific application to implement a wear leveling scheme for that application.

In some embodiments, one or more devices having accelerator virtualization and/or tiered memory in accordance with example embodiments of the disclosure may be used to implement memory and/or accelerator dis-aggregation, for example, through the use of single-level and/or multi-level switching.

<FIG> illustrates an embodiment of a system with dis-aggregation in accordance with example embodiments of the disclosure. The system illustrated in <FIG> may include a host <NUM> connected to a first switch 578a and a second switch 578b through interconnects 556a and 556b, respectively. The first switch 578a may be connected to a first device 500a and a first memory 580a through interconnects 556c and 556d, respectively. The second switch 578b may be connected to a second device 500b and a second memory 580b through interconnects 556e and 556f, respectively. In some embodiments, the first and second switches 578a and 578b, and interconnects 556a-556f (which may collectively form an interconnect fabric) may be implemented with PCle switches and/or interconnects using CXL protocol.

The first and second devices 500a and 500b may be implemented, for example, with one or more devices having accelerator virtualization and/or tiered memory in accordance with example embodiments of the disclosure.

The system illustrated in <FIG> may enable resources such as the first and second switches 578a and 578b and the first and second memories 580a and 580b to be taken online and/or offline based, for example, on current processing and/or memory demands. One or more of the resources may be time multiplexed, for example, across domains. Moreover, one or more of downstream ports (e.g., host root ports 582a and 582b) and or switches (e.g., switches 578a and 578b) may be virtualized, for example, for use with multiple virtual NPUs <NUM>-<NUM>,. , <NUM>-N illustrated in <FIG>. In some embodiments, one or more of the devices 500a and 500b may be implemented as a multi-headed device with multiple ports for use, for example, may multiple virtual accelerators.

The system illustrated in <FIG> may also include a fabric manager that may perform one or more functions such as device discovery, virtual switch creation and/or management, binding virtual ports to physical ports, and/or the like. In some embodiments, the fabric manager may be located at the host <NUM> and implemented, for example, with one or more device drivers. In some embodiments, the fabric manager may be implemented in a sideband configuration, for example, with a system management bus (SMBus). Although the embodiment illustrated in <FIG> is shown with two switches, two devices, and two memories, any number of components, may be used with any number of switches, interconnects, and/or the like using any level of switching.

A device having accelerator virtualization and/or tiered memory in accordance with example embodiments of the disclosure may be used to implement various types of parallelism. For example, data parallelism may involve splitting data for a workload across multiple accelerator devices, each of which may have the same model. Depending on the implementation details, data parallelism may be compute bound, especially for large models. However, a device having a tiered memory system in accordance with example embodiments of the disclosure may accommodate a large model, for example, in a capacity-enhanced memory of the second memory tier. Moreover, the use of a cache in the second memory tier may compensate for potential latency issues with the capacity-enhanced memory.

With model parallelism, portions of a model may be split across multiple accelerators, and the same data may be processed by each accelerator. For example, if a model is split in half between NPU0 and NPU1, all or a portion of the data may be processed first by NPU0, then the same data may be processed by NPU1. Model parallelism may be used, for example, for allreduce algorithms and/or all-to-all (A2A) communication where one or more (sometimes all) NPUs communicate with one or more (sometimes all) other NPUs after each epoch. A device having accelerator virtualization in accordance with example embodiments of the disclosure may accommodate model parallelism, for example, by enabling each of multiple virtual accelerators to handle a portion of a model.

In some embodiments, choosing between data parallelism and model parallelism may involve one or more tradeoffs. For example, with a large model, data parallelism may be difficult to implement because the entire model may be replicated and stored at each accelerator. Moreover, with data parallelism, the data may need to be synchronized for training a DL model. For example, weights may be synchronized during training because each accelerator may be working on the same model with different training data, so the system may synchronized data and determine one or more average values for weights for each epoch. With model parallelism, fewer memory and/or storage resources may be used to store only a portion of the model at each accelerator, and training synchronization issues may be avoided. However, depending on the implementation details, communication between accelerators may increase.

Depending on the implementation details, the device <NUM> illustrated in <FIG> may reduce or eliminate tradeoffs, for example, when implementing model parallelism with allreduce and/or all-to-all communication primitives. For example, the use of a coherent interface such as CXL may enable multiple NPUs (e.g., multiple NPUs <NUM> at different instances of device <NUM> and/or multiple virtual NPUs <NUM>-<NUM>,. , <NUM>-N within a single device <NUM>) to exchange data at a smaller granularity (e.g., using CXL flits). Thus, the allreduce engine <NUM> may be configured as a station that may gather updates from various NPUs using CXL primitives and/or launching data lines as needed by the various NPUs. In some embodiments, the allreduce engine <NUM> may also be configured to implement message passing interface (MPI) send and/or receive primitives over the interconnect <NUM> and/or network <NUM>. Depending on the implementation details, this may enable scaling out the system to multiple accelerators over the network <NUM> with model parallelism. Moreover, CXL may enable coherence across multiple accelerators distributed within a device (e.g., as one or more NPUs <NUM> and/or one or more virtual NPUs <NUM>-<NUM>,. , <NUM>-N), a server, a rack, across a network, and/or the like. Thus, in some embodiments, the allreduce engine <NUM> may implement a combination of MPI and CXL, which may clear CXL requests to gather updates for use by the device <NUM> and/or generate updates that other devices may use for scaling up to multiple NPUs. Depending on the implementation details, this may enable the allreduce engine <NUM> to exchange and/or schedule tensors efficiently, e.g., using CXL semantics.

<FIG> illustrates an embodiment of a graph, and a method for global partitioning of the graph, in accordance with example embodiments of the disclosure. The graph <NUM> illustrated in <FIG> may include twelve vertices <NUM> indicated as v<NUM> through v<NUM> connected by edges (x,y) where x and y may indicate vertices connected by the edge. With directed edges, x and y may indicate the origin and destination vertices, respectively. With undirected edges, x and y may be unordered. Thus, edge (<NUM>,<NUM>) indicates the edge between vertex v<NUM> and vertex v<NUM>. (To prevent obscuring the drawing, not all edges may be labeled.

The graph <NUM> may be globally partitioned by splitting it at dashed lines 686a and 686b into three portions 684a (which may include vertices v<NUM> through v<NUM>), 684b (which may include vertices v<NUM> through v<NUM>), and 684c (which may include vertices v<NUM> through v<NUM>). The graph <NUM> may be partitioned, for example, by an application running on a host and implementing an ML framework such as TensorFlow, PyTorch, and/or the like.

The application may partition the vertices into three accelerator devices <NUM> indicated as dev<NUM> through dev<NUM>. Specifically, the application may partition vertices the first portion 684a of the graph into dev<NUM>, the second portion 684b of the graph into dev<NUM>, and the third portion 684c of the graph into dev<NUM>. The application may partition the vertices based, for example, on one or more parameters of the graph elements (e.g., vertices and/or edges) such as memory to be used, computation time, and/or the like, and one or more parameters of the accelerator devices <NUM> such as compute resources, memory, interconnects, and/or the like.

Although the graph illustrated in <FIG> may include twelve vertices and <NUM> edges and may be partitioned into three portions, in other embodiments, any number of vertices, edges, and/or partitions in any configuration, may be used.

<FIG> illustrates a local scheduling operation in accordance with example embodiments of the disclosure. The embodiment illustrated in <FIG> may be used, for example, to schedule graph processing operations for the portions 684a, 684b, and 684c of the graph using the accelerator devices <NUM> indicated as dev<NUM>, dev<NUM>, and dev<NUM>, respectively, into which they have been partitioned as illustrated in <FIG>.

Referring to <FIG>, edges for which both vertices are within one device (e.g., edge (<NUM>,<NUM>)) may be scheduled for processing with the corresponding device (e.g., dev<NUM>). Edges between vertices that span two devices (e.g., edge (<NUM>,<NUM>)) may be scheduled for processing by one or both of the corresponding devices. For example, the device dev<NUM> containing the destination vertex v<NUM> may process edge (<NUM>,<NUM>).

<FIG> illustrates an embodiment of a device virtualization method in accordance with example embodiments of the disclosure. Although the embodiment illustrated in <FIG> may be used for any type of application, depending on the implementation details, some embodiments may be especially useful for implementing device virtualization for multiple large DNNs. For purposes of illustrating the principles of the disclosure, the method illustrated in <FIG> uses the graph and graph partitions illustrated in <FIG> and <FIG>. However, the principles may be implemented with any other graphs, models, and/or the like, and partitions thereof.

Referring to <FIG>, for purposes of illustration a graph <NUM> may be implemented in a manner similar to the graph <NUM> illustrated in <FIG>, but any other graph, model, and/or the like, may be used. In this embodiment, operations <NUM>, <NUM>, and <NUM> may be performed by a host, and operation <NUM> may be performed by a device, but in other embodiments, the operations may be distributed differently. The host and device may be implemented, for example, using any of the host and/or devices disclosed herein, including those described with respect to <FIG>, <FIG>, and/or <FIG>. The graph <NUM> may be implemented, for example, by an application <NUM> such as an ML framework running on the host.

Referring to <FIG>, at operation <NUM> (Part <NUM>. ), during a configuration process prior to runtime, a device may present itself to the host as multiple virtual NPUs, each having one or more resources including, for example, one or more processor cores, memory (e.g., tiered or non-tiered memory), interconnects, and/or the like. In some embodiments, one or more interconnects to the host may be implemented as inter-device and/or intra-device.

At operation <NUM> (Part <NUM>), the host may annotate graph inputs and/or outputs with one or more parameters such as an amount of memory to be used, a computation time, and/or the like.

At operation <NUM> (Part <NUM>), the host may partition the graph <NUM> into devices dev<NUM>, dev<NUM>, and dev<NUM>, based on one or more parameters of the virtual NPUs such as compute resources (e.g., processor cores), memory, interconnects, and/or the like. In some embodiments, the host may attempt to match the one or more parameters of the graph portions such as memory to be used, computation time, and/or the like, with corresponding parameters of the virtual accelerator devices such as compute resources (e.g., processor cores), memory, interconnects, and/or the like.

Some example values for parameters of the graph portions may be as follows:.

At operation <NUM> (Part <NUM>. ), the device (e.g., the device implementing the virtual NPUs) may extract one or more operational parameters from the graph partitions provided by the host such as memory usage, task dependencies, timing information for one or more graph partitions, and/or the like for use at runtime. In some embodiments, the device may set up one or more process address spaces for each virtual device and partition. Each process address space may have a process address space identifier (PASID). In the example illustrated in <FIG>, the device may set up a process address space PASID0, PASID1, and PASID2 for virtual accelerators dev<NUM>, dev<NUM>, and dev<NUM>, respectively in the physical accelerator <NUM>. In some embodiments, PASID may be implemented in the context of CXL, for example, with CPU dedicated timesharing, shared virtual machines (VMs), and/or shared single root input and/or output virtualization (SR-IOV).

<FIG> illustrates an embodiment of a workflow for device virtualization method in accordance with example embodiments of the disclosure. The workflow illustrated in <FIG> may be used, for example, with the methods illustrated in <FIG> and <FIG>.

Referring to <FIG>, at operation <NUM>, one or more user applications running, for example, at a host, may create one or more graph representations. At operation <NUM>, the one or more applications may implement the one or more graph representations using an ML framework. At operation <NUM>, the one or more applications may partition a graph into one or more portions and map the one or more portions into one or more NPUs (e.g., virtual NPUs), for example, for memory performance (e.g., optimization) based on operational parameters such as task dependency, peak memory capacity, minimum synchronization overhead, and/or the like.

<FIG> and <FIG>, which collectively form <FIG>, illustrate an embodiment of a memory scheduling method in accordance with example embodiments of the disclosure. Although the embodiment illustrated in <FIG> may be used for any type of application, depending on the implementation details, some embodiments may be especially useful for implementing memory scheduling for multiple large DNNs. For purposes of illustrating the principles of the disclosure, the method illustrated in <FIG> uses the graph and graph partitions illustrated in <FIG> and <FIG>. However, the principles may be implemented with any other graphs, models, and/or the like, and partitions thereof.

Referring to <FIG>, for purposes of illustration, a graph <NUM> may be implemented in a manner similar to the graph <NUM> illustrated in <FIG>, but any other graph, model, and/or the like, may be used. In this embodiment, operations <NUM>, <NUM>, <NUM>, and <NUM> may be performed by a host. The graph <NUM> may be implemented, for example, by an application <NUM> such as an ML framework running on the host, and a memory graph generator <NUM> may be implemented on the host as well. In other embodiments, however, the operations may be distributed differently, for example, between a host and a device. A host and device may be implemented, for example, using any of the host and/or devices disclosed herein, including those described with respect to <FIG>, <FIG>, and/or <FIG>.

Referring to <FIG>, at operation <NUM> (Part <NUM>. ), during a configuration process prior to runtime, a device (e.g., a CXL Type-<NUM> device) may present itself to the host as multiple virtual NPUs, each having one or more resources including, for example, one or more processor cores, memory (e.g., tiered or non-tiered memory), interconnects, and/or the like.

At operation <NUM> (Part <NUM>), the host (e.g., using the memory graph generator <NUM>) may generate a clustered graph <NUM> with memory capacities and/or duration information. The clustered graph <NUM> may include data groups (indicated as Group <NUM>, Group <NUM>, and Group <NUM>) distributed between virtual NPUs (indicated as VNPU1, VNPU2, and VNPU3). In the embodiment illustrated in <FIG>, VNPU1 (Group <NUM>) and VNPU2 (Group <NUM>) may be implemented on a first physical device (Device <NUM>), and VNPU3 (Group <NUM>) may be implemented on a second physical device (Device <NUM>). The memory usages and timing values shown for the clustered graph <NUM> illustrated in <FIG> may be implemented on the two physical devices as shown in <FIG>.

Referring to <FIG>, the NPU memory <NUM>-<NUM> for Device <NUM>, which may function as the working memory for VNPU1 and VNPU2, may be implemented, for example, with the bandwidth-enhanced memory <NUM> illustrated in <FIG>. The tiered memory <NUM>-<NUM> for Device <NUM> may be implemented, for example, with the latency-enhanced memory <NUM> illustrated in <FIG>. Similarly, the NPU memory <NUM>-<NUM> for Device <NUM>, which may function as the working memory for VNPU3, may be implemented, for example, with the bandwidth-enhanced memory <NUM> illustrated in <FIG>, and the tiered memory <NUM>-<NUM> for Device <NUM> may be implemented, for example, with the latency-enhanced memory <NUM> illustrated in <FIG>.

Referring again to <FIG>, the Group <NUM> data initially may be loaded into the NPU memory <NUM>-<NUM> for Device <NUM>. As VNPU1 processes the Group <NUM> data, it may gradually swap out data to the tiered memory <NUM>-<NUM>, for example, on a priority basis as shown at (<NUM>). Also, as VNPU1 processes the Group <NUM> data, it may transfer data, for example, the 15GB block, at time T8 as shown at (<NUM>), to the NPU memory <NUM>-<NUM> of Device <NUM> where it may be processed in Group <NUM> by VNPU3. Similarly, after processing the 13GB block of data, VNPU3 may transfer the data, to the NPU memory <NUM>-<NUM> of Device <NUM> as shown at (<NUM>), where it may be processed as part of Group <NUM> by VNPU2. In the intervening time, VNPU2 at Device <NUM> may begin processing the Group <NUM> data, for example, as soon as it has a minimum amount of data and/or memory to process as shown at (<NUM>).

<FIG> illustrates an embodiment of an allreduce operation method in accordance with example embodiments of the disclosure. The embodiment illustrated in <FIG> may be implemented, for example, using any of the hosts and/or devices disclosed herein, including those described with respect to <FIG>, <FIG>, and/or <FIG>.

Referring to <FIG>, the method may be implemented using a host <NUM>, an interconnect controller (e.g., a CXL controller) <NUM>, and one or more devices <NUM> having one or more accelerators. For purposes of illustration, the embodiment illustrated in <FIG> is implemented with eight accelerators <NUM>-<NUM> through <NUM>-<NUM>, however, the principles may be applied with any number of devices. The accelerators <NUM>-<NUM> through <NUM>-<NUM> may be implemented with physical accelerators, virtual accelerators, or any combination thereof. In some embodiments, the embodiment illustrated in <FIG> may be implemented partially or entirely with the allreduce engine <NUM> illustrated in <FIG>.

Referring to <FIG>, the method may proceed as shown by the solid and dashed arrows wherein the different shading indicates different datasets and the ⊕ operator indicates an allreduce operation.

<FIG> illustrates an embodiment of a method for processing data with a model in accordance with example embodiments of the disclosure. The method may begin at operation <NUM>. At operation <NUM>, the method may partition a model into a first portion and a second portion. For example, the method may partition the model based on one or more parameters of the model and/or one or more parameters of a first virtual accelerator and a second virtual accelerator. At operation <NUM>, the method may store the first portion of the model in a memory of a device. At operation <NUM>, the method may store the second portion of the model in the memory of the device. In some embodiments, the memory may be implemented as a tiered memory. At operation <NUM>, the model may perform, by the first virtual accelerator at the device, a first operation using the first portion of the model. At operation <NUM>, the method may perform, by the second virtual accelerator at the device, a second operation using the second portion of the model. The method may end at operation <NUM>.

The embodiment illustrated in <FIG>, as well as all of the other embodiments described herein, are example operations and/or components. In some embodiments, some operations and/or components may be omitted and/or other operations and/or components may be included. Moreover, in some embodiments, the temporal and/or spatial order of the operations and/or components may be varied. Although some components and/or operations may be illustrated as individual components, in some embodiments, some components and/or operations shown separately may be integrated into single components and/or operations, and/or some components and/or operations shown as single components and/or operations may be implemented with multiple components and/or operations.

Any of the functionality described herein, including any of the host functionality, device functionally, and/or the like, including that described with respect to <FIG> for example, an accelerator, IO engine, allreduce engine, compression and/or decompression logic, memory switch, memory manager, preprocessor, virtualization logic, memory access pattern analyzer, DCOH, and/or the like, may be implemented with hardware, software, or any combination thereof including combinational logic, sequential logic, one or more timers, counters, registers, state machines, CPLDs, FPGAs, ASICs, CPUs including CISC processors such as x86 processors and/or RISC processors such as ARM processors, GPUs, NPUs, TPUs, and/or the like, executing instructions stored in any type of memory. In some embodiments, one or more components may be implemented as a system-on-chip (SOC).

Claim 1:
A device (<NUM>, <NUM>, <NUM>, 500a, 500b, <NUM>) comprising:
an interconnect interface (<NUM>, <NUM>, <NUM>);
a memory system (<NUM>, <NUM>, <NUM>) comprising:
one or more first type memory devices (<NUM>, <NUM>) coupled to the interconnect interface (<NUM>, <NUM>, <NUM>) to receive first data and configured to function as working memory for an accelerator (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) of the device (<NUM>, <NUM>, <NUM>, 500a, 500b, <NUM>);
one or more second type memory devices (<NUM>, <NUM>) coupled to the interconnect interface (<NUM>, <NUM>, <NUM>) to receive second data, and wherein the first type relates to high-bandwidth and the second type relates to high-capacity;
the accelerator (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) coupled to the one or more first type memory devices (<NUM>, <NUM>) and the one or more second type memory devices (<NUM>, <NUM>) and configured to perform an operation using the first data and the second data; and
a memory switch (<NUM>) arranged to configure one or more connections between the one or more first and second type memory devices (<NUM>, <NUM>; <NUM>, <NUM>) and the accelerator (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>), wherein the memory switch (<NUM>) is programmable to configure the one or more first type memory devices (<NUM>, <NUM>) and the one or more second type memory devices (<NUM>, <NUM>), and programmable to configure a specific amount of cache to the accelerator (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>).