Patent Description:
At least some known special purpose accelerators, e.g., and without limitation, deep-learning accelerators, include an interconnected set of processing cores within a multicore processing device, where each core uses a local memory, or scratchpad, for quick access to its working set, and can access a global memory to periodically, or continually, replenish the contents of the scratchpad. The scratchpad is typically implemented as a static random access memory (SRAM) array with low-latency and high bandwidth capability, while the global memory is typically implemented as a dynamic random access memory (DRAM) array external to the chip containing the multiple processor cores. A periodicity of data replenishment of the scratchpad is determined by the relationship between the size of the working set of the application and the physical capacity of the scratchpad. The larger the scratchpad, the less often the replenishment. Document <CIT> discloses a system with buffer areas in multi-channel memory that are accessed via a crossbar interconnect by multiple processing elements, grouping the buffer areas assigned to each processing element and partitioning the crossbar interconnect based on the grouping of the buffer areas.

A system and method are provided for augmenting capacity of the main on-chip scratchpad for a processing core.

In one aspect, a memory system configured to augment a capacity of a plurality of main scratchpads for a plurality of respective processing cores is presented, The memory system includes a global memory device coupled to a plurality of processing elements. The global memory device is positioned external to a chip on which the plurality of processing elements reside. The memory system also includes at least one main scratchpad coupled to the of the plurality of processing elements and the global memory device. The memory system further includes a plurality of auxiliary scratchpads coupled to the plurality of processing elements and the global memory device. At least a portion of the plurality of auxiliary scratchpads are configured as a unitary multichannel device. Accordingly, alleviating bandwidth constraints is implemented through devices that overcome the capacity constraints of existing memory-intensive systems, such as those implemented in neural network designs.

In another aspect, a method of assembling a memory system configured to augment a capacity of a plurality of main scratchpads for a plurality of respective processing cores is presented. The method includes positioning a plurality of processing elements on a chip and coupling a global memory device to the plurality of processing elements. The global memory device is positioned external to the chip. The method also includes coupling at least one main scratchpad to at least one processing element of the plurality of processing elements and the global memory device. The method further includes coupling a plurality of auxiliary scratchpads to the plurality of processing elements and the global memory device. At least a portion of the plurality of auxiliary scratchpads are configured as a unitary multichannel device. Accordingly, alleviating bandwidth constraints is implemented through devices that overcome the capacity constraints of existing memory-intensive systems, such as those implemented in neural network designs.

In yet another aspect, a computer system configured to augment a capacity of a plurality of main scratchpads for a plurality of respective processing cores is presented. The computer system includes a plurality of processing devices positioned on a chip. Each processing device of the one or more processing devices includes one or more processing elements. The computer system also includes at least one main scratchpad coupled to the one or more processing elements. The computer system further includes a global memory device coupled to the plurality of processing devices. The global memory device is positioned external to the chip and the global memory device is coupled to the at least one main scratchpad. The computer system also includes one or more auxiliary scratchpads coupled to the plurality of processing devices and the global memory device. At least a portion of the plurality of auxiliary scratchpads are configured as a unitary multichannel device. Accordingly, alleviating bandwidth constraints in implemented through devices that overcome the capacity constraints of existing memory-intensive systems, such as those implemented in neural network designs.

The present Summary is not intended to illustrate each aspect of every implementation of, and/or every embodiment of the present disclosure. These and other features and advantages will become apparent from the following detailed description of the present embodiment(s), taken in conjunction with the accompanying drawings.

The drawings included in the present application are incorporated into, and form part of, the specification. They illustrate embodiments of the present disclosure and, along with the description, serve to explain the principles of the disclosure. The drawings are illustrative of certain embodiments and do not limit the disclosure.

While the present disclosure is amenable to various modifications and alternative forms, specifics thereof have been shown by way of example in the drawings and will be described in detail. It should be understood, however, that the intention is not to limit the present disclosure to the particular embodiments described.

Aspects of the present disclosure relate to augmenting a capacity of the main on-chip scratchpad for a processing core. While the present disclosure is not necessarily limited to such applications, various aspects of the disclosure may be appreciated through a discussion of various examples using this context.

It will be readily understood that the components of the present embodiments, as generally described and illustrated in the Figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following details description of the embodiments of the apparatus, system, method, and computer program product of the present embodiments, as presented in the Figures, is not intended to limit the scope of the embodiments, as claimed, but is merely representative of selected embodiments.

Reference throughout this specification to "a select embodiment," "at least one embodiment," "one embodiment," "another embodiment," "other embodiments," or "an embodiment" and similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases "a select embodiment," "at least one embodiment," "in one embodiment," "another embodiment," "other embodiments," or "an embodiment" in various places throughout this specification are not necessarily referring to the same embodiment.

The illustrated embodiments will be best understood by reference to the drawings, wherein like parts are designated by like numerals throughout. The following description is intended only by way of example, and simply illustrates certain selected embodiments of devices, systems, and processes that are consistent with the embodiments as claimed herein.

As used herein, "facilitating" an action includes performing the action, making the action easier, helping to carry the action out, or causing the action to be performed. Thus, by way of example and not limitation, instructions executing on one processor might facilitate an action carried out by semiconductor processing equipment, by sending appropriate data or commands to cause or aid the action to be performed. Where an actor facilitates an action by other than performing the action, the action is nevertheless performed by some entity or combination of entities.

In general, as the robustness of modern computing systems increase, additional memory strategies are implemented to reduce access times where performance is limited by the speed and efficiency of operations that use the memory bus. Use of memory controllers reduces access times somewhat, but typically do not overcome all of the limitations of a relatively remote memory bus from the processing devices. Merely increasing the amount of accessible memory does not alleviate the extended access times. In addition, defining a number of separate channels between the memory and the processing devices tends to decrease latencies; however, the use of software features to manage the data flow through the various channels may tend to add latencies back in. Other known solutions include the use of three-dimensional stacked DRAM (3D DRAM) to localizing communication between columns of memory elements and their underlying processor cores to improve access times. However, some of these solutions use a multiplexing device that somewhat increases the access times for certain operations.

More specifically, at least some known special purpose accelerators, e.g., and without limitation, deep-learning accelerators, include an interconnected set of processing cores within a multicore processing device, where each core uses a local memory, or scratchpad, for quick access to its working set, and can access a global memory device to periodically, or continually, replenish the contents of the scratchpad. The working set is the memory that a process requires in a given time interval. For example, with respect to the working set size, for a <NUM> gigabyte (GB) main memory system, only <NUM> megabytes (MB) may be necessary for a particular application for each second of operation; therefore, for such a scenario, <NUM> MB is the working set size.

The scratchpad is typically implemented as a static random access memory (SRAM) array directly positioned on the respective chip, or processing core, with low-latency and high bandwidth capability, where the close proximity of the main scratchpad and the processing elements facilitates the low-latency aspects of the on-chip devices. In contrast, the global memory is typically implemented as a dynamic random access memory (DRAM) array external to the chip containing the multiple processor cores. A periodicity of data replenishment of the scratchpad is determined by the relationship between the working set size of the application and the physical capacity of the scratchpad. Generally, scratchpads are distinguished from memory cache where the scratch pads tend to be more deterministic with respect to the data contents resident thereon, thereby facilitating fetch and prefetch operations for only certain data. Accordingly, for certain processing requirements, scratchpads are preferable to cache.

At least some known efforts to reduce these overheads and improve the utilization of the processing resources typically includes creating a memory hierarchy, where additional tiers of memory are directly inserted between the main scratchpad and the global memory device. These tiers are typically implemented as caches of the main global memory and replenishment is achieved by moving contents from one memory tier to the next memory tier in succession. When the movement between the tiers is implemented in hardware, special circuits are provided to determine which locations in each of the tiers are to be moved, and when they should be moved to maximize the utilization of processing resources. The additional hardware is used to both implement the tiers of memory on the same chip as the processor, and for controlling the transmission between the tiers, and this hardware can be quite complex. Moreover, the additional hardware components are typically not aware of the intentions of the programs executing on the processors, and hence unable to maximize processor utilization. They also consume valuable real estate on the processor chip, that could otherwise have been devoted to increasing the processing power of the chip.

In addition, some known efforts to reduce these overheads and improve the utilization of the processing resources typically includes increasing the size of the main scratchpad. In general, the larger the main scratchpad, the less often data replenishment is required. More frequent replenishment facilitates increasing the overhead in processing, both in terms of power to transmit data back and forth, and in terms of time needed for these transmissions. Moreover, the time needed for this transmission invariably causes an idling of the processing resources, and hence underutilization of the processor. However, merely increasing the size of the main scratchpad tends to contravene efforts to reduce the size of the chip on which the main scratchpad is resident. Therefore, the opportunity cost of implementing the additional tiers on-chip as described above is also a similar drawback of simply increasing the size of the local scratchpad memory. Additionally, increasing the size of the main scratchpad also increases the latency of access to the main scratchpad even for applications with smaller working sets that may not need the larger scratchpad memory resources.

Accordingly, there is a need for alleviating bandwidth constraints through devices that overcome the capacity constraints of existing memory-intensive systems, such as those implemented in neural network designs.

Referring to <FIG>, a block schematic diagram is presented illustrating a memory system architecture <NUM> (sometimes referred to as the memory system <NUM> herein) including multichannel direct random access memory (DRAM) features, in accordance with some embodiments of the present disclosure. In some embodiments, the memory system architecture <NUM> is integrated on a single die. In some embodiments, at least a portion of the memory system architecture <NUM> includes at least a portion of a neural network accelerator chip <NUM>; however, the memory system architecture <NUM> is not limited to such implementation and is configured to be employed in any implementation and equipment architecture that enables operation of the memory system architecture <NUM> as described herein.

In some embodiments, the memory system architecture <NUM> includes a global memory device <NUM>. The global memory device <NUM>, in at least some embodiments, is implemented as a dynamic random access memory (DRAM) device. In some embodiments, the global memory device <NUM> is implemented as a DRAM array. In some embodiments, the global memory device <NUM> is on the same die as the neural network accelerator chip <NUM>. In some embodiments, the global memory device <NUM> is integrated into a chip separate from the neural network accelerator chip <NUM> (as shown in <FIG>).

In one or more embodiments, the neural network accelerator chip <NUM> includes a memory controller <NUM> that is communicatively and operably coupled to the global memory device <NUM>. The memory controller <NUM> is a device that manages the flow of data transmitted to, and transmitted from, the global memory device <NUM>. In at least some embodiments, the neural network accelerator chip <NUM> includes an on-chip interconnect <NUM> that is communicatively and operably coupled to the memory controller <NUM> through an interconnect conduit <NUM>. In addition, in some embodiments, the neural network accelerator chip <NUM> includes a plurality of processing cores <NUM> extending from processing core <NUM>-<NUM> through <NUM>-n, where the processing core <NUM>-n-<NUM> is read as "the nth-minus-<NUM>" processing core, etc., and where the variable "n" has any value that enables operation of the neural network accelerator chip <NUM>, including, without limitation, values of <NUM> through <NUM>. Each processing core <NUM> is communicatively coupled to the on-chip interconnect <NUM> through an on-chip interconnect channel <NUM> (shown in dashed phantom and only one labeled in <FIG> for clarity). The processing cores <NUM> are discussed further with respect to <FIG>.

Furthermore, the neural network accelerator chip <NUM> includes a multichannel memory array <NUM>, herein referred to as the multichannel auxiliary DRAM array <NUM>. The multichannel auxiliary DRAM array <NUM> is configured to augment the global memory device <NUM>. In some embodiments, the multichannel auxiliary DRAM array <NUM> is a unitary device (as shown in <FIG>); however, in some embodiments the multichannel auxiliary DRAM array <NUM> is distributed into a plurality of auxiliary DRAM devices to allow for other chip architectural elements to be positioned most appropriately. The multichannel auxiliary DRAM array <NUM> includes a plurality of auxiliary DRAM elements, or channels, herein referred to as auxiliary scratchpads <NUM> extending from an auxiliary scratchpad <NUM>-<NUM> to an auxiliary scratchpad <NUM>-n. Accordingly, the neural network accelerator chip <NUM> includes a plurality of multilateral segments in the form of auxiliary scratchpads <NUM>, where at least a portion of the plurality of auxiliary scratchpads <NUM> are configured as a unitary multichannel device. i.e., one or more multichannel auxiliary DRAM arrays <NUM>.

In some embodiments, each auxiliary scratchpad <NUM> is directly, communicatively, and operably coupled to a corresponding processing core <NUM> through an auxiliary scratchpad channel <NUM> (only two shown and labeled for clarity, i.e., <NUM>-<NUM> and <NUM>-n), e.g., the auxiliary scratchpad <NUM>-<NUM> is coupled to the processing core <NUM>-<NUM> and the auxiliary scratchpad <NUM>-n is coupled to the processing core <NUM>-n. In some embodiments, in contrast to the one-to-one relationship between the auxiliary scratchpads <NUM> and the processing cores <NUM>, at least some of the processing cores <NUM> are coupled to a plurality of auxiliary scratchpads <NUM>. In some embodiments, the respective auxiliary scratchpad channel <NUM> cooperates with the associated auxiliary scratchpad <NUM> to define the respective auxiliary DRAM channels. In some embodiments, the global memory device <NUM> and the multichannel auxiliary DRAM array <NUM> are implemented physically alongside each other in a parallel configuration. Therefore, the multichannel auxiliary DRAM array <NUM>, and the individual auxiliary scratchpads <NUM>, are located off the respective processing cores (or chips) <NUM>, thereby allowing the processing core chip <NUM> designers to use the unused portions of the processing core <NUM> for other features. Moreover, the use DRAM for the auxiliary scratchpads <NUM> facilitates taking advantage of the density of DRAM technology as opposed to the less dense SRAM or embedded-DRAM technologies. In some embodiments, such use of DRAM technology allows the auxiliary scratchpads <NUM> to contain the working set of, for example, and without limitation, large deep learning inference applications targeted to natural language understanding.

Referring to <FIG>, a block schematic diagram is presented illustrating an enlarged view of a portion <NUM> of the memory system architecture <NUM>, and more specifically, a portion of the neural network accelerator chip <NUM> (shown in <FIG>), in accordance with some embodiments of the present disclosure, where reference to <FIG> is continued. In one or more embodiments, each processing core <NUM> includes a main on-chip scratchpad <NUM>, referred to herein as the main scratchpad <NUM>, coupled to the on-chip interconnect <NUM> through the respective on-chip interconnect channel <NUM>. In some embodiments, the main scratchpad <NUM> at least partially defines the on-chip interconnect channel <NUM>. In some embodiments, the main scratchpad <NUM> is configured as a static random access memory (SRAM) array directly positioned on the respective chip, or processing core <NUM>, with low-latency and high bandwidth capabilities, where the close proximity of the main scratchpad <NUM> and the processing elements <NUM> facilitates the low-latency aspects of the on-chip devices of the memory system architecture <NUM>. In some embodiments, the main scratchpad <NUM> has any memory architecture that enables operation of the memory system architecture <NUM>, including the neural network accelerator chip <NUM>, including, without limitation, a DRAM device.

Further, in some embodiments, each processing core <NUM> includes a channel controller <NUM> that is coupled to the respective auxiliary scratchpad <NUM> through the respective auxiliary scratchpad channel <NUM>. The channel controller <NUM> is configured to manage the transmission of signals from the respective auxiliary scratchpad <NUM> to the processing element <NUM>. In some embodiments, the channel controller <NUM> at least partially defines the auxiliary scratchpad channel <NUM>. Also, in some embodiments, each processing core <NUM> includes a multiplexor (MUX) <NUM> that is configured to channel selected signals from the respective main scratchpad <NUM> and the respective auxiliary scratchpad <NUM> (through the channel controller <NUM>) to the processing element <NUM> for the desired processing operations. Accordingly, the respective auxiliary scratchpads <NUM> are physically positioned off the chip with the processing core <NUM> and mapped to the respective processing elements <NUM> in parallel with the respective main scratchpad <NUM>.

In some embodiments, by implementing the memory features of the auxiliary scratchpad <NUM> to augment the memory features of the main scratchpad <NUM>, the contents of the memory as well as the orchestration of the movement of the contents back and forth from the respective processor cores <NUM> can be orchestrated completely in software, thus allowing the utilization of the processing resources to approach the theoretical maximum. In some embodiments, a reduction in the total energy consumption of the processing cores <NUM> may be realized due to the improvement in performance of the neural network accelerator chip <NUM> as a result of the memory system augmentation as described herein.

In at least some embodiments, the auxiliary scratchpad <NUM> is configured to facilitate the processing of static tensors and dynamic tensors within a neural network, such as those neural networks found within deep learning platforms, including, without limitation, the neural network accelerator chip <NUM> at least partially shown in <FIG> and <FIG>. In addition, the auxiliary scratchpads described herein are suitable for use in the broader range of artificial intelligence platforms, including, without limitation, machine learning platforms. In general, a tensor represents a multi-dimensional array containing elements of a single data type configured to be used for arbitrary numeric computation. Therefore, in at least some embodiments, the deep learning tensors are configured as one or more matrices as represented using n-dimensional arrays (where the term "n-dimensional" as used with respect to the tensors is not associated with the variable number of processing cores <NUM> and auxiliary scratchpads <NUM> described herein). In one non-limiting example, numerical data in the form of <NUM> values is stored in a single array implemented as a single, contiguous block in memory as a <NUM>-by-<NUM>-by-<NUM> tensor, one value after another, where three dimensions is also non-limiting. In some embodiments, the tensors are static with respect to the shape, i.e., the number of dimensions and the extent of each dimension in the matrix do not change over time. One non-limiting example for the values for a static tensor include model weights that are read-only values and are employed during the definition (sometimes referred to as the inference time, where the model remains fixed) of a respective computational graph of nodes in the neural network corresponding to operations or variables therein. In contrast, in some embodiments, the tensors are dynamic, i.e., the values in these dynamic tensors are not fixed, and these dynamic tensors are produced as intermediate or final outputs, where the lifetime of these dynamic tensors is not necessarily for the full execution of the program. In some embodiments, for example, and without limitation, the matrix configuration with respect to the number of dimensions and the extent of each dimension within the array is subject to change during the course of execution of the respective computational graph.

In at least some embodiments, the auxiliary scratchpads <NUM> are configured to store static deep learning tensors including, without limitation, weighting values for facilitating the operation of the respective neural network. In contrast, the main scratchpad <NUM> is configured to retain the dynamic tensors that are also employed for facilitating the operation of the respective neural network, e.g., and without limitation, node activations. In some embodiments, the auxiliary scratchpads <NUM> are configured with a read bandwidth that is lower than the read bandwidth for the main scratchpad <NUM>, where the more limited static contents of the auxiliary scratchpads <NUM> do not require the larger read bandwidths typically found in the main scratchpad <NUM>. In addition, at least partially due to the aforementioned properties of the data stored in the auxiliary scratchpads <NUM>, the write bandwidth of the auxiliary scratchpads <NUM> is lower than the read bandwidth of the auxiliary scratchpads <NUM>. Accordingly, in such embodiments, the read/write bandwidths of the auxiliary scratchpads <NUM> do not need to be as large as the read/write bandwidths of the main scratchpads <NUM>.

In some embodiments, the read/write bandwidths of the auxiliary scratchpads <NUM> are enlarged to accommodate storing dynamic tensors as necessary for those embodiments of the memory system <NUM> and the neural network accelerator chip <NUM> that require such configurations.

In one or more embodiments, the neural network accelerator chip <NUM> includes a one-to-one relationship between the auxiliary scratchpads <NUM> and the processing cores <NUM>. In some embodiments, the neural network accelerator chip <NUM> includes a more than one-to-one relationship between the auxiliary scratchpads <NUM> and the processing cores <NUM>, where one limiting factor includes the amount of remaining space availability with respect to the physical landscape of the neural network accelerator chip <NUM>. Therefore, in some embodiments, a plurality of auxiliary scratchpads <NUM> are coupled to the respective MUX <NUM> through one of individual respective channel controllers <NUM> or a unitary multichannel controller (not shown). In some embodiments, the processing element <NUM> is configured to accommodate any number of auxiliary scratchpads <NUM>, including, without limitation, <NUM> through <NUM> auxiliary scratchpads <NUM>.

Referring to <FIG>, a block schematic diagram is presented illustrating a memory system architecture <NUM> (sometimes referred to as the memory system <NUM> herein) including multichannel DRAM features, in accordance with some embodiments of the present disclosure. In many embodiments, the memory system <NUM> is similar to the memory system <NUM> (shown in <FIG> and <FIG>), with one difference being the addition of a global channel <NUM> (only one labeled). Also, referring to <FIG>, a block schematic diagram is presented illustrating an enlarged view of a portion <NUM> of the memory system architecture <NUM> shown in <FIG>, in accordance with some embodiments of the present disclosure. Similarly numbered components in <FIG>, <FIG>, <FIG>, and <FIG> are similarly named with similar functionality.

The global channel <NUM> is communicatively coupled to the on-chip interconnect <NUM>, that is communicatively coupled to the global memory device <NUM>. In addition, the global memory channel <NUM> is communicatively and operably coupled to each of the processing cores <NUM>, thereby directly coupling the global memory device <NUM> to the processing cores <NUM>. Such direct coupling facilitates those instances where there are data calls by the respective processing element <NUM> for data resident within other portions of the memory system <NUM>.

In at least some embodiments, the global channel <NUM> is coupled to the channel controller <NUM> to facilitate managing the flow of information in and out of the mux <NUM> for the processing element <NUM>. In some embodiments, the global channel <NUM> is coupled to the processing element <NUM> through any mechanism that enables operation of the memory system <NUM> as described herein.

Referring to <FIG>, a block schematic diagram is presented illustrating a memory system architecture <NUM> (sometimes referred to as the memory system <NUM> herein) including multichannel DRAM features, in accordance with some embodiments of the present disclosure. Similarly numbered components in <FIG>, <FIG>, <FIG>, and <FIG> are similarly named with similar functionality, and reference to <FIG>, <FIG>, <FIG>, and <FIG> continues.

In at least some embodiments, at least a portion of the memory system <NUM> includes one or more neural network accelerator chips in a manner similar to the network accelerator chips <NUM>/<NUM>. In some embodiments, the memory system <NUM> includes an on-chip interconnect <NUM> that is substantially similar to the on-chip interconnects <NUM>/<NUM>. In at least some embodiments, the memory system <NUM> includes a global memory device (that is substantially similar to the global memory devices <NUM>/<NUM>) that is communicatively coupled to the on-chip interconnect <NUM> through a memory controller (that is substantially similar to the memory controllers <NUM>/<NUM>) through the interconnect conduit <NUM> (that is substantially similar to the interconnect conduits <NUM>/<NUM>), where the global memory device and the memory controller are not shown in <FIG> for clarity.

In one or more embodiments, the memory system <NUM> includes a plurality of chiplets <NUM>, i.e., a first chiplet <NUM>-<NUM>, a second chiplet <NUM>-<NUM>, etc., through an mth chiplet <NUM>-m, where the variable "m" has any value that enables operation of the memory system <NUM>, including, without limitation, values of <NUM> through <NUM>. In some embodiments, the chiplets <NUM> reside on the neural network accelerator chip (not shown in <FIG>) as described above. Each chiplet <NUM> includes one or more processing cores <NUM>, where, for example, a non-limiting number of two processing cores <NUM>-<NUM>-<NUM> and <NUM>-<NUM>-<NUM> are shown for the first chiplet <NUM>-<NUM> in <FIG>. Similarly, the second chiplet <NUM>-<NUM> includes, for example, a non-limiting number of two processing cores <NUM>-<NUM>-<NUM> and <NUM>-<NUM>-<NUM>, and the mth chiplet <NUM>-m includes, for example, a non-limiting number of two processing cores <NUM>-m-<NUM> and <NUM>-m-<NUM>. In some embodiments, each chiplet <NUM> includes any number of processing cores <NUM> that enables operation of the memory system <NUM> as described herein, including, without limitation, <NUM> through <NUM> processing cores <NUM>. Each processing core <NUM> is communicatively coupled to the on-chip interconnect <NUM> through an on-chip interconnect channel <NUM> (shown in dashed phantom and only one labeled in <FIG> for clarity). The processing cores <NUM> are discussed further with respect to <FIG>.

Further, in at least some embodiments, each chiplet <NUM> includes one or more auxiliary DRAM elements, or channels, herein referred to as auxiliary scratchpads <NUM>. As shown in <FIG>, each chiplet <NUM> includes one or more auxiliary scratchpads <NUM>, where, for example, a non-limiting number of two auxiliary scratchpads <NUM>-<NUM>-<NUM> and <NUM>-<NUM>-<NUM> are shown for the first chiplet <NUM>-<NUM> in <FIG>. Similarly, the second chiplet <NUM>-<NUM> includes, for example, a non-limiting number of two auxiliary scratchpads <NUM>-<NUM>-<NUM> and <NUM>-<NUM>-<NUM>, and the mth chiplet <NUM>-m includes, for example, a non-limiting number of two auxiliary scratchpads <NUM>-m-<NUM> and <NUM>-m-<NUM>. In some embodiments, each chiplet <NUM> includes any number of auxiliary scratchpads <NUM> that enables operation of the memory system <NUM> as described herein, including, without limitation, <NUM> through <NUM> auxiliary scratchpads <NUM>.

In some embodiments, the plurality of auxiliary scratchpads <NUM> for each chiplet <NUM> are individual elements positioned the respective chiplets <NUM> such that each chiplet <NUM> is a unitary device. In contrast, in some embodiments, the plurality of chiplets <NUM> are formed as a separate multichannel auxiliary DRAM array <NUM> that is coupled across the full set of chiplets <NUM> (as shown in <FIG> with the single-dashed/single-dotted lines). Accordingly, in such embodiments, the multichannel auxiliary DRAM array <NUM> includes a plurality of multilateral segments in the form of auxiliary scratchpads <NUM>, where at least a portion of the plurality of auxiliary scratchpads <NUM> are configured as a unitary multichannel device, e.g., each chiplet <NUM> and the multichannel auxiliary DRAM array <NUM> are such devices.

In some embodiments, each auxiliary scratchpad <NUM> is directly, communicatively, and operably coupled to a corresponding processing core <NUM> through an auxiliary scratchpad channel <NUM> (only one shown and labeled for clarity), e.g., the auxiliary scratchpad <NUM>-<NUM>-<NUM> is coupled to the processing core <NUM>-<NUM>-<NUM> and the auxiliary scratchpad <NUM>-m-<NUM> is coupled to the processing core <NUM>-m-<NUM>. In some embodiments, in contrast to the one-to-one relationship between the auxiliary scratchpads <NUM> and the processing cores <NUM>, at least some of the processing cores <NUM> are coupled to a plurality of auxiliary scratchpads <NUM>. In some embodiments, the respective auxiliary scratchpad channel <NUM> cooperates with the associated auxiliary scratchpad <NUM> to define the respective auxiliary DRAM channels.

Referring to <FIG>, a block schematic diagram is presented illustrating an enlarged view of a portion <NUM> of the memory system architecture <NUM> shown in <FIG>, in accordance with some embodiments of the present disclosure. In many embodiments, the portion <NUM> of the memory system <NUM> is substantially similar to the portion <NUM> of the memory system <NUM>. Similarly numbered components in <FIG> and <FIG> are similarly named with similar functionality.

Referring to <FIG>, a block schematic diagram is presented illustrating a memory system architecture <NUM> including multichannel DRAM features, in accordance with some embodiments of the present disclosure. Also referring to <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>, in at least some embodiments, the memory system architecture <NUM> includes at least one logic processor die <NUM>, where the number one is non-limiting. In some embodiments, the logic processor die <NUM> includes a plurality of processing cores <NUM> that are substantially similar to the processing cores <NUM>, <NUM>, and <NUM>. The logic processor die <NUM> includes any number of processing cores <NUM> that enables operation of the memory systems <NUM>, <NUM>, <NUM>, and <NUM> as described herein including, without limitation, <NUM> through <NUM>, where <NUM> units of processing cores <NUM> are shown in <FIG>, and <NUM> processing cores <NUM> are labeled from processing core <NUM>-<NUM> to <NUM>-<NUM>.

In addition, in some embodiments, the memory system architecture <NUM> includes a plurality of DRAM dies <NUM>. In some embodiments, the DRAM dies <NUM> are similar to the multichannel auxiliary DRAM arrays <NUM>, <NUM>, or <NUM>. The memory system architecture <NUM> includes any number of DRAM dies <NUM> that enables operation of the memory systems <NUM>, <NUM>, <NUM>, and <NUM> as described herein including, without limitation, <NUM> through <NUM>, where <NUM> units of DRAM dies <NUM> are shown in <FIG>, i.e., DRAM dies <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, and <NUM>-<NUM>. As shown, the memory system architecture <NUM> defines a three-dimensional (3D) stacked configuration. In some embodiments, the number of stacked DRAM dies <NUM> is subject to environmental conditions, including, without limitation, thermal considerations with respect to heat generation and removal. In addition, the number of stacked DRAM dies <NUM> chosen is subject to chip design considerations with respect to positioning of the various components thereon, and the practical limitations based on modern chip manufacturing techniques and the structural strength requirements of the DRAM dies <NUM>.

In one or more embodiments, each DRAM die <NUM> includes a plurality of auxiliary scratchpads <NUM> that are substantially similar to the auxiliary scratchpad <NUM>. In some embodiments the plurality of auxiliary scratchpads <NUM> are substantially similar to the auxiliary scratchpads <NUM> and <NUM>. In some embodiments, each DRAM die <NUM> includes any number of auxiliary scratchpads <NUM> that enables operation of the memory systems <NUM>, <NUM>, <NUM>, and <NUM> as described herein including, without limitation, <NUM> through <NUM>, where <NUM> units of auxiliary scratchpads <NUM> are shown for each of the four DRAM dies <NUM> in <FIG>, and <NUM> processing cores <NUM> are labeled from processing core <NUM>-<NUM> to <NUM>-<NUM>. In some embodiments, electric connections through the stack are facilitated through a plurality of through silicon vias (TSVs) <NUM>. Communications between the processing cores <NUM> and the auxiliary scratchpads <NUM> are discussed with respect to <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>.

As shown in <FIG>, in some embodiments, each processor core <NUM> is communicatively and operably coupled to four auxiliary scratchpads <NUM>. For example, the processor core <NUM>-<NUM> is coupled to the auxiliary scratchpads <NUM>-<NUM>-<NUM>, <NUM>-<NUM>-<NUM>, <NUM>-<NUM>-<NUM>, and <NUM>-<NUM>-<NUM>. In some embodiments, the processor cores <NUM> are coupled to any number of auxiliary scratchpads <NUM> that enables operation of the memory system architecture <NUM> as described herein.

Referring to <FIG>, a block schematic diagram is presented illustrating a memory system architecture <NUM> including multichannel DRAM features, in accordance with some embodiments of the present disclosure. The memory system architecture <NUM> is similar to the memory system architecture <NUM> (shown in <FIG>, also continued to be referred to); however, rather than a stacked configuration, the memory system architecture <NUM> is a parallel configuration. Therefore, the memory system architecture <NUM> includes at least one logic processor die <NUM> that is similar to the logic processor die <NUM>. In addition, in some embodiments, the memory system architecture <NUM> includes one or more DRAM dies <NUM>, where two DRAM dies <NUM>-<NUM> and <NUM>-<NUM> are shown. The DRAM dies <NUM> are similar to the DRAM dies <NUM>. Communications between the processing cores <NUM> and the auxiliary scratchpads <NUM> are discussed with respect to <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>.

Referring now to <FIG>, a block schematic diagram is provided illustrating a computing system <NUM> that may be used in implementing one or more of the methods, tools, and modules, and any related functions, described herein (e.g., using one or more processor circuits or computer processors of the computer), in accordance with some embodiments of the present disclosure. In some embodiments, the major components of the computer system <NUM> may comprise one or more CPUs <NUM>, a memory subsystem <NUM>, a terminal interface <NUM>, a storage interface <NUM>, an I/O (Input/Output) device interface <NUM>, and a network interface <NUM>, all of which may be communicatively coupled, directly or indirectly, for inter-component communication via a memory bus <NUM>, an I/O bus <NUM>, and an I/O bus interface unit <NUM>.

The computer system <NUM> may contain one or more general-purpose programmable central processing units (CPUs) <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-<NUM>, <NUM>-N, herein collectively referred to as the CPU <NUM>. In some embodiments, the computer system <NUM> may contain multiple processors typical of a relatively large system; however, in other embodiments the computer system <NUM> may alternatively be a single CPU system. Each CPU <NUM> may execute instructions stored in the memory subsystem <NUM> and may include one or more levels of on-board cache.

System memory <NUM> may include computer system readable media in the form of volatile memory, such as random access memory (RAM) <NUM> or cache memory <NUM>. Computer system <NUM> may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system <NUM> can be provided for reading from and writing to a non-removable, non-volatile magnetic media, such as a "hard drive. " Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), or an optical disk drive for reading from or writing to a removable, non-volatile optical disc such as a CD-ROM, DVD-ROM or other optical media can be provided. In addition, memory <NUM> can include flash memory, e.g., a flash memory stick drive or a flash drive. Moreover, the global memory devices, the main scratchpads, and the auxiliary scratchpads as described herein are included as a portion of the described suite of memory devices. Memory devices can be connected to memory bus <NUM> by one or more data media interfaces. The memory <NUM> may include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of various embodiments.

Although the memory bus <NUM> is shown in <FIG> as a single bus structure providing a direct communication path among the CPUs <NUM>, the memory subsystem <NUM>, and the I/O bus interface <NUM>, the memory bus <NUM> may, in some embodiments, include multiple different buses or communication paths, which may be arranged in any of various forms, such as point-to-point links in hierarchical, star or web configurations, multiple hierarchical buses, parallel and redundant paths, or any other appropriate type of configuration. Furthermore, while the I/O bus interface <NUM> and the I/O bus <NUM> are shown as single respective units, the computer system <NUM> may, in some embodiments, contain multiple I/O bus interface units <NUM>, multiple I/O buses <NUM>, or both. Further, while multiple I/O interface units are shown, which separate the I/O bus <NUM> from various communications paths running to the various I/O devices, in other embodiments some or all of the I/O devices may be connected directly to one or more system I/O buses.

In some embodiments, the computer system <NUM> may be a multi-user mainframe computer system, a single-user system, or a server computer or similar device that has little or no direct user interface, but receives requests from other computer systems (clients). Further, in some embodiments, the computer system <NUM> may be implemented as a desktop computer, portable computer, laptop or notebook computer, tablet computer, pocket computer, telephone, smart phone, network switches or routers, or any other appropriate type of electronic device.

It is noted that <FIG> is intended to depict the representative major components of an exemplary computer system <NUM>. In some embodiments, however, individual components may have greater or lesser complexity than as represented in <FIG>, components other than or in addition to those shown in <FIG> may be present, and the number, type, and configuration of such components may vary.

One or more programs/utilities <NUM>, each having at least one set of program modules <NUM> may be stored in memory <NUM>. The programs/utilities <NUM> may include a hypervisor (also referred to as a virtual machine monitor), one or more operating systems, one or more application programs, other program modules, and program data. Each of the operating systems, one or more application programs, other program modules, and program data or some combination thereof, may include an implementation of a networking environment. Programs <NUM> and/or program modules <NUM> generally perform the functions or methodologies of various embodiments.

Referring to FIG. 7A, a flowchart is provided illustrating a process <NUM> for assembling a computer system, such as the computer system <NUM> (see <FIG>). Also referring to <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>, the process <NUM> includes positioning <NUM> a plurality of processing elements <NUM>/<NUM>/<NUM> on a chip <NUM>/<NUM>. The process <NUM> also includes coupling <NUM> the global memory device <NUM>/<NUM> to the plurality of processing elements <NUM>/<NUM>/<NUM>, where the global memory device <NUM>/<NUM> is positioned external to the chip <NUM>/<NUM>. The process <NUM> further includes coupling <NUM> at least one main scratchpad <NUM>/<NUM>/<NUM> to the at least one processing element of the plurality of processing elements <NUM>/<NUM>/<NUM> and the global memory device <NUM>/<NUM>. The process <NUM> also includes coupling <NUM> a plurality of auxiliary scratchpads <NUM>/<NUM>/<NUM> to the respective processing elements <NUM>/<NUM>/<NUM> and the global memory device <NUM>/<NUM>. At least a portion of the plurality of auxiliary scratchpads <NUM>/<NUM>/<NUM> are configured as a unitary multichannel device, i.e., either the multichannel auxiliary DRAM arrays <NUM>/<NUM>/<NUM> and the chiplets <NUM>.

The process <NUM> also includes coupling <NUM> each auxiliary scratchpad <NUM>/<NUM>/<NUM> to a respective processing core <NUM>/<NUM>/<NUM>, thereby defining <NUM> a plurality of auxiliary scratchpad channels <NUM>/<NUM>/<NUM>. In some embodiments, coupling a channel controller <NUM>/<NUM>/<NUM> to the respective auxiliary scratchpad <NUM>/<NUM>/<NUM> further defines <NUM> the respective auxiliary scratchpad channel <NUM>/<NUM>/<NUM>. In some embodiments, a neural network accelerator chip <NUM>/<NUM> is fabricated <NUM>. In some embodiments, the plurality of chiplets <NUM> are fabricated <NUM>. Both the neural network accelerator chip <NUM>/<NUM> and the chiplets <NUM> include the main scratchpad <NUM>/<NUM>/<NUM>, the processing element <NUM>/<NUM>/<NUM>, and the auxiliary scratchpad <NUM>/<NUM>/<NUM>.

The embodiments as disclosed and described herein are configured to provide an improvement to computer technology. Materials, operable structures, and techniques as disclosed herein can provide substantial beneficial technical effects. Some embodiments may not have all of these potential advantages and these potential advantages are not necessarily required of all embodiments. By way of example only, and without limitation, one or more embodiments may provide enhancements of the operation of memory systems through the addition of dedicated auxiliary scratchpad memories to the individual processor cores.

In at least some embodiments as described herein, the enhancement of the memory systems includes higher bandwidths since the off-chip memory bandwidths are typically limited by the respective packaging features. In addition, as the computing performance of modern computing systems increases, the memory bandwidth is also increased in proportion to the compute gains for balanced system performance. In addition, the use of off-chip memory access is relatively power-intensive as compared to on-chip memory access. Therefore, the embodiments described herein, including the 3D stacked embodiments, with the close proximity of the processor cores and the auxiliary DRAM scratchpads on the same chip facilitate decreasing the power consumption of the associated computer system. Furthermore, the 3D stacked DRAM embodiments described herein facilitate greater memory density than on-die SRAM; therefore, the embodiments described herein facilitate a much greater memory capacity in a smaller form factor thereby facilitating larger memory capacities. Moreover, with respect to the form factors, the embodiments described herein facilitate reducing reliance on memory packages external to the chip, thereby reducing, and in some cases, eliminating the need for external memory packages.

In one or more embodiments described herein, additional benefits are attained when executing smaller batch cases more quickly through the respective computing systems for those operations where amortization of the costs associated with importing the various weighting values is not feasible, i.e., such costs are not recoverable.

The present disclosure may be a system, a method, and/or a computer program product at any possible technical detail level of integration.

Computer readable program instructions for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, and procedural programming languages, such as the "C" programming language or similar programming languages. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

Claim 1:
A memory system configured to augment a capacity of a plurality of main scratchpads for a plurality of respective processing cores, the memory system comprising:
a global memory device coupled to a plurality of processing elements, wherein the global memory device is positioned external to a chip on which the plurality of processing elements reside; the memory system being characterized in that it comprises :
at least one main scratchpad coupled to at least one processing element of the plurality of processing elements and the global memory device; and
a plurality of auxiliary scratchpads coupled to the plurality of processing elements and the global memory device, wherein at least a portion of the plurality of auxiliary scratchpads are configured as a unitary multichannel device.