Patent ID: 12236134

DETAILED DESCRIPTION

Overview

Processing in memory (PIM) components are embedded within a memory module to enable data to be obtained from a memory and processed entirely within the memory module. By doing so, the PIM components alleviate memory performance and energy bottlenecks by moving memory-intensive computations closer to memory. Further, the PIM components reduce data transfer latency and improve overall computer performance, as compared to standard computer architectures which access off-chip memory systems.

Generally, a dynamic random access memory (DRAM) includes a plurality of banks where data is stored. More specifically, the banks of a DRAM include memory arrays that are organized into rows and columns of a grid, such that data is maintained in individual cells of the grid. In order to access a bank of the DRAM (e.g., to load data from the bank of the DRAM for further processing by a PIM component and/or a host processing unit or to store data that has been processed by the PIM component and/or the host processing unit to the bank of the DRAM), a particular row of the bank where data is to be loaded from and/or stored to is opened. Further, in order to open a new row in a bank of the DRAM, a row that is currently open in the bank is closed. Therefore, in order to access a new DRAM row, a memory controller is employed to issue a precharge command to close a currently open DRAM row and an activate command to open the new DRAM row.

Notably, there is a considerable amount of overhead for opening a DRAM row (e.g., referred to as row activate overhead time) and closing a DRAM row (e.g., referred to as row precharge overhead time). In order to hide this overhead, standard computer architectures (e.g., those which access data from off-chip memory systems) utilize bank-level parallelism. Bank-level parallelism is a degree to which multiple banks of a memory are concurrently occupied with servicing memory commands. To enable bank-level parallelism, standard or conventional memory controllers include complex logic and advanced queueing techniques to re-order memory commands to keep multiple banks busy with servicing the memory commands and to maximize row buffer locality.

In contrast to conventional memory commands, PIM commands often depend on data maintained in registers (e.g., temporary data storage) of a PIM component and are broadcast to multiple banks in parallel. Due to these differences, PIM commands are issued in program order (e.g., without being re-ordered) to preserve functional correctness. As a result, bank-level parallelism is typically not achievable without adding significant memory controller complexity for conventional PIM-enabled systems, leading to low PIM throughput, and decreased computational efficiency of the conventional systems.

Techniques for bank-level parallelism with processing in memory are described herein. In accordance with the described techniques, a DRAM is organized into a plurality of blocks, each including a bank assigned to a first group, a bank assigned to a second group, and a PIM component. Further, the memory controller receives a plurality of PIM commands which are scheduled (e.g., by a compiler) in streams of commands that alternate between access to the first group of banks and the second group of banks. For example, the PIM commands are received by the memory controller in the following order: a first stream of PIM commands that access the first group of banks, a second stream of PIM commands that access the second group of banks, a third stream of PIM commands that access the first group of banks, and so forth. An identification command is inserted (e.g., by the compiler) after each stream of PIM commands. The identification command identifies the next row in a respective group of banks that is subsequently accessed by the PIM components. By way of example, an identification command is inserted after a stream of PIM commands that access a first row of the first group of banks, the identification command identifies a second row of the first group of banks, and the second row of the first group of banks is next accessed by the PIM components. Notably, the identification command instructs the memory controller to preemptively close a row that is no longer accessed by the PIM commands and/or activate a row that will be accessed by the PIM commands in the future without causing any column operations to be performed.

Given an identification command inserted after a stream of commands that access a first row in the first group of banks and identifying a second row in the first group of banks, the memory controller is instructed to schedule a precharge command to close the first row and an activate command to open the second row. In one or more implementations, the memory controller schedules the activate command in parallel with execution of a subsequent stream of PIM commands, which cause the processing in memory components to perform operations that access the second group of banks. Additionally or alternatively, the memory controller schedules the precharge command in parallel with execution of the subsequent stream of PIM commands, which cause the processing in memory components to perform operations that access the second group of banks.

By doing so, the memory controller110overlaps row activate overhead time and/or row precharge overhead time in one group of banks with execution of PIM commands that access a different group of banks. Therefore, the described techniques enable processing of PIM commands faster than conventional PIM-enabled systems, leading to higher PIM throughput and increased computational efficiency.

In some aspects, the techniques described herein relate to a computing device including a memory that includes a first bank and a second bank, a processing in memory component embedded in the memory, and a memory controller to receive a plurality of commands for execution by the processing in memory component, the plurality of commands including a first stream of commands which cause the processing in memory component to perform operations that access the first bank and a second stream of commands which cause the processing in memory component to perform operations that access the second bank, identify a next row of the first bank that is to be accessed by the processing in memory component after the first stream of commands, and schedule, an activate command to open the next row of the first bank in parallel with execution of the second stream of commands.

In some aspects, the techniques described herein relate to a computing device, wherein the next row is identified based on an identification command of the plurality of commands which identifies the next row to be accessed by the processing in memory component without causing the processing in memory component to perform operations that access the next row.

In some aspects, the techniques described herein relate to a computing device, wherein the memory is organized into a plurality of blocks, each block including a bank assigned to a first group, a bank assigned to a second group, and a respective processing in memory component shared among the banks included in a respective block.

In some aspects, the techniques described herein relate to a computing device, wherein the first bank is assigned to the first group, the second bank is assigned to the second group, and the processing in memory component is shared among the first bank and the second bank.

In some aspects, the techniques described herein relate to a computing device, wherein the activate command causes banks assigned to the first group to open the next row in parallel, and the second stream of commands cause the respective processing in memory components to perform operations that access banks assigned to the second group in parallel.

In some aspects, the techniques described herein relate to a computing device, wherein the memory controller is further configured to inspect rows accessed by the plurality of commands and scheduling the activate command based on the inspection.

In some aspects, the techniques described herein relate to a computing device, wherein the memory controller is further configured to predict, using a prediction policy, the next row of the first bank to be accessed by the processing in memory component and schedule the activate command to open the next row based on the prediction.

In some aspects, the techniques described herein relate to a computing device, wherein the memory controller is further configured to update the prediction policy based on whether a row of the first bank that is opened based on the predicting is a row of the first bank that is accessed next by the processing in memory component.

In some aspects, the techniques described herein relate to a computing device, wherein the memory controller is further configured to receive one or more hints indicative of at least one of an amount of data that is to be processed by the processing in memory component in executing the plurality of commands or a number of data structures involved in the plurality of commands, and predict the next row of the first bank to be accessed by the processing in memory component based on the one or more hints.

In some aspects, the techniques described herein relate to a computing device, wherein the plurality of commands include the first stream of commands which cause the processing in memory component to perform operations that access a first row of the first bank, and an identification command which identifies the next row of the first bank.

In some aspects, the techniques described herein relate to a computing device, wherein the memory controller is further configured to identify a first set of the second stream of commands having a combined execution time that is equal to or greater than an overhead time for closing a row of the memory.

In some aspects, the techniques described herein relate to a computing device, wherein the memory controller is further configured to schedule, based on the identification command, a precharge command to close the first row of the first bank in parallel with execution of the first set of the second stream of commands by the processing in memory component.

In some aspects, the techniques described herein relate to a computing device, wherein the activate command is scheduled after the first set of the second stream of commands and causes the first bank to open the next row in parallel with execution of a second set of the second stream of commands by the processing in memory component.

In some aspects, the techniques described herein relate to a system including a memory that includes a first bank and a second bank, a processing in memory component embedded in the memory, a core to issue commands for execution by the processing in memory component, the commands including an identification command that identifies a next row of the first bank that is to be accessed by the processing in memory component, and a memory controller to receive the commands from the core, and schedule, based on the identification command, a precharge command to close a first row of the first bank in parallel with execution of a first set of the commands which cause the processing in memory component to perform operations that access the second bank.

In some aspects, the techniques described herein relate to a system, wherein the memory is a dynamic random access memory organized into a plurality of blocks, each block including a bank assigned to a first group, a bank assigned to a second group, and a respective processing in memory component shared among the banks included in a respective block, and the first bank, the second bank, and the processing in memory component comprise one block of the dynamic random access memory.

In some aspects, the techniques described herein relate to a system, wherein the memory controller is further configured to schedule, based on the identification command, an activate command to open the next row of the first bank in parallel with execution of a second set of the commands which cause the processing in memory component to perform operations that access the second bank

In some aspects, the techniques described herein relate to a system, wherein the memory controller is further configured to delay issuance of the activate command until after an overhead time for closing the first row has elapsed.

In some aspects, the techniques described herein relate to a method including compiling a program to generate a plurality of commands for execution by a processing in memory component embedded in a memory, the memory including a first bank and a second bank, the plurality of commands including an identification command that identifies a next row of the first bank that is to be accessed by the processing in memory component and a set of the commands which cause the processing in memory component to perform operations that access the second bank, and transmitting the plurality of commands to a memory controller, the identification command instructing the memory controller to close a first row of the first bank and open the next row of the first bank in parallel with execution of the set of the commands.

In some aspects, the techniques described herein relate to a method, wherein transmitting the plurality of commands includes transmitting a first stream of the commands which cause the processing in memory component to perform operations that access the first bank followed by a second stream of the commands including the set of the commands which cause the processing in memory component to perform operations that access the second bank.

In some aspects, the techniques described herein relate to a method, the method further including calculating a number of commands to include in the first stream of commands and the second stream of commands, the number of commands having a combined execution time that is equal to or greater than an overhead time for closing a row of the memory and opening an additional row of the memory.

FIG.1is a block diagram of a non-limiting example system100having a host with a core and a memory controller coupled to a memory module having a memory and one or more processing in memory components embedded in the memory module. In particular, the system100includes a host102and a memory module104. In this example, the host102includes a compiler106which runs on a core108and a memory controller110. The memory module104includes a memory112organized into a plurality of blocks114. Each block114includes a first bank116, a second bank118, and a processing in memory (PIM) component120. In one or more implementations, the compiler106compiles a program to generate a plurality of PIM commands122for execution by the PIM component(s)120.

In accordance with the described techniques, the host102and the memory module104are coupled to one another via a wired or wireless connection. The core108and the memory controller110are also coupled to one another via one or more wired or wireless connections. Example wired connections include, but are not limited to, buses (e.g., a data bus), interconnects, traces, and planes. Examples of devices in which the system100is implemented include, but are not limited to, servers, personal computers, laptops, desktops, game consoles, set top boxes, tablets, smartphones, mobile devices, virtual and/or augmented reality devices, wearables, medical devices, systems on chips, and other computing devices or systems.

The host102is an electronic circuit that performs various operations on and/or using data in the memory112. Examples of the host102include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), a field programmable gate array (FPGA), an accelerated processing unit (APU), and a digital signal processor (DSP). The core108is a processing unit that reads and executes commands (e.g., of a program), examples of which include to add and to move data. Although one core108is depicted in the example system100, in variations, the host102includes more than one core108, e.g., the host102is a multi-core processor.

In one or more implementations, the memory module104is a circuit board (e.g., a printed circuit board), on which the memory112is mounted and includes the PIM component120. In variations, one or more integrated circuits of the memory112are mounted on the circuit board of the memory module104, and the memory module104includes one or more PIM components120. Examples of the memory module104include, but are not limited to, single in-line memory module (SIMM), dual in-line memory module (DIMM), high-bandwidth memory (HBM), and TransFlash memory module.

The memory112is a device or system that is used to store information, such as for immediate use in a device, e.g., by the core108of the host102and/or by the PIM component120. In one or more implementations, the memory112corresponds to semiconductor memory where data is stored within memory cells on one or more integrated circuits. In at least one example, the memory112corresponds to or includes volatile memory, examples of which include random-access memory (RAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), and static random-access memory (SRAM). Alternatively or in addition, the memory112corresponds to or includes non-volatile memory, examples of which include flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM). Thus, the memory112is configurable in a variety of ways that support bank-level parallelism for processing in memory without departing from the spirit or scope of the described techniques.

The memory controller110is a digital circuit that manages the flow of data to and from the memory112. By way of example, the memory controller110includes logic to read and write to the memory112and interface with the PIM component120, e.g., to provide PIM commands122to the PIM component120for processing by the PIM component120. The memory controller110also interfaces with the core108. For instance, the memory controller110receives commands from the core108which involve accessing the memory112and provides data to the core108, e.g., for processing by the core108. In one or more implementations, the memory controller110is communicatively located between the core108and the memory module104, and the memory controller110interfaces with both the core108and the memory module104.

In one or more implementations, the host102generates the PIM commands122by running the compiler106, and transmits the PIM commands122, via the memory controller110, to the memory module104. Broadly, the PIM component120includes one or more in-memory processors and is configured to process the PIM commands122. For example, the one or more in-memory processors of the PIM component120processes the PIM commands122using data stored in the memory112. Processing in memory using in-memory processors contrasts with standard computer architectures which obtain data from memory, communicate the data to the core108of the host102, and process the data using the core108rather than the PIM component120.

In various scenarios, the data produced by the core108as a result of processing the obtained data is written back to the memory112, which involves communicating the produced data over the pathway from the core108to the memory112. In terms of data communication pathways, the core108of the host102is further away from the memory112than the PIM component120. As a result, these standard computer architectures suffer from increased data transfer latency, particularly when the volume of data transferred between the memory112and the host102is large, which can also decrease overall computer performance. Thus, the PIM component120increases computer performance while reducing data transfer latency as compared to standard computer architectures that which access off-chip memory systems. Further, the PIM component120alleviates memory performance and energy bottlenecks by moving one or more memory-intensive computations closer to the memory112.

In one or more implementations, the memory112is a dynamic random access memory, and includes a plurality of banks (e.g., the first bank116and the second bank118) where data is stored. In particular, the banks are organized into one or more memory arrays (e.g., grids), which include rows and columns such that data is stored in individual cells of the memory arrays.

In order to access data maintained in the memory112organized in this way (e.g., to load data from a bank of the memory112for further processing by the PIM component120, and/or to store data that has been processed by the PIM component to a bank of the memory112), a particular row of the bank where data is to be loaded from and/or stored to is opened. As used herein, “opening” a row of a bank causes data maintained in the row to be loaded into a row buffer of the bank. From the row buffer, data maintained in a row and column address of the open row can be read to registers (e.g., temporary data storage) maintained by the PIM component120. In one or more implementations, the PIM component120executes the PIM commands122using data residing in the registers. Further, data produced by the PIM component120(e.g., as a result of processing the data residing in the registers) can be written from the registers back to the row buffer of the bank. In order to open a new row of the bank, a row that is currently open in the row buffer of the bank is closed. As used herein, “closing” a row clears the row buffer and causes the data that is currently maintained in the row buffer to be written back to a corresponding row of the bank.

The memory controller110is configured to schedule and issue DRAM commands that are utilized to execute the PIM commands122. The DRAM commands include precharge commands and activate commands. Given a set of PIM commands122which cause the PIM component120to perform operations that access a row of the first bank116(e.g., “load” operations which load data from the row of the first bank116to the registers and/or “store” operations which store data that has been processed by the PIM component120to the row of the first bank116), the memory controller110is configured to issue a precharge command to close a previously opened row in the first bank116as well as an activate command to open the row of the first bank116that is accessed by the set of PIM commands122.

The amount of time that it takes the memory112to service the DRAM commands is referred to herein as DRAM overhead time. The DRAM overhead time, for instance, includes row precharge overhead time (e.g., the time it takes for the memory112to close a row) and row activate overhead time (e.g., the time it takes for the memory112to open a row). To hide DRAM overhead time, standard computer architectures (e.g., those which communicate data to the core108of the host102for processing, rather than the PIM component120) utilize bank-level parallelism. Broadly, a system will incur a decrease in performance due to low bank-level parallelism when banks sit idle waiting for other banks to service memory commands. In contrast, a system improves performance due to high bank-level parallelism when all or a majority of banks are concurrently occupied with servicing memory commands.

To achieve such parallelism, a standard memory controller typically employs logic to re-order memory commands to keep multiple banks busy servicing the memory commands and to maximize row buffer locality. To do so, the standard memory controller inspects row addresses associated with enqueued commands to schedule the commands in look-ahead fashion, e.g., to schedule commands that meet all dependencies without waiting for a previous command to complete. This scheduling policy implemented by the standard memory controller tracks all commands, their timing requirements, and all dependencies between commands. Accordingly, standard computer architectures require complex logic and advanced queuing techniques to support bank-level parallelism.

In contrast to those memory commands, the PIM commands122are broadcast to multiple banks in parallel, and as such, are subject to different memory constraints. Further, the PIM commands122often depend on data maintained in the registers of the PIM component. Due to this, dependencies between the PIM commands122are not determined by simply inspecting row addresses associated with the PIM commands122. Instead, the dependencies between the PIM commands122are determinable by additionally inspecting the registers accessed by the PIM commands122. In one or more implementations, the memory controller110does not include the complex logic utilized to inspect the registers accessed by the PIM commands122. In one or more variations, the PIM commands are issued in order (e.g., without being re-ordered) in order to preserve functional correctness.

As a result, bank-level parallelism is typically not achievable without adding significant memory controller complexity for conventional PIM-enabled systems which include multiple banks that share a PIM component. While a PIM component of a conventionally configured system executes one or more PIM commands that access one bank, for instance, other banks that service the PIM component often sit idle, e.g., without servicing memory commands. Further, when a new row is subsequently accessed, such conventionally configured systems typically wait for row precharge overhead time (e.g., to close a previously opened row) and row activate overhead time (e.g., to open the new row) before servicing a subsequent PIM command that accesses the new row. This leads to low PIM throughput and performance degradation. These problems are further exacerbated because PIM-enabled systems often utilize more frequent row accesses (e.g., rows are opened and closed more frequently) in comparison to standard systems.

To overcome the drawbacks of conventionally configured systems having PIM components, techniques to enable bank-level parallelism for processing in memory are used. In accordance with the described techniques, the memory112is organized into a plurality of blocks114. In one or more implementations, for example, each block114includes a same number of banks and each of the banks in a given block114is assigned to a different group. Thus, in the illustrated example involving two banks per block114, each block114includes a bank (e.g., the first bank116) assigned to a first group and an additional bank (e.g., the second bank118) assigned to a second group. In variations, the memory112can include any number of blocks114and any number of banks per block114.

In accordance with the described techniques, the compiler106compiles a program to generate a plurality of PIM commands122for execution by the PIM component120. In one or more implementations, the compiler106schedules the PIM commands122in streams of PIM commands which alternate between access to the first bank116and access to the second bank118—or that are interleaved across more banks in variations. For example, the compiler106schedules a stream of PIM commands122that access the first bank116, followed by a second stream of PIM commands that access the second bank118, followed by a third stream of PIM commands that access the first bank116, and so forth.

In implementations, the compiler106inserts an identification command after each stream of PIM commands122that access a particular bank. An identification command, for instance, is inserted after a stream of commands that access a row of a particular bank, and the identification command identifies a next row of the particular bank that is subsequently accessed by the PIM component120. By way of example, an identification command is inserted after a first stream of PIM commands122that access a first row of the first bank116, and the identification command identifies a next row of the first bank116that is to be accessed by the PIM component120. Furthermore, an additional identification command is inserted after a second stream of PIM commands that access a first row of the second bank118, and the additional identification command identifies the next row of the second bank118that is to be accessed by the PIM component120. Notably, the identification command identifies the next row of a respective bank that is to be accessed by the PIM component120without causing the PIM component to perform operations (e.g., load operations and store operations) that access the next row.

The memory controller110receives the PIM commands122in the order determined by the compiler106and enqueues the PIM commands122(e.g., in a PIM queue) in the determined order. Further, the memory controller110issues the PIM commands in the determined order to the memory112for execution by the PIM component120, e.g., without re-ordering the PIM commands. As previously described, the memory controller110also schedules DRAM commands that are utilized to execute the PIM commands122. Notably, the DRAM commands, in contrast to the PIM commands122, are scheduled in look-ahead fashion. In other words, the memory controller can schedule DRAM commands which meet all dependencies without waiting for a previous DRAM command to complete.

In one example, the memory controller110receives a first stream of PIM commands that access a first row of the first bank116, an identification command that identifies a next row of the first bank116that is to be accessed by the PIM component120, and a second stream of PIM commands that access the second bank118. The identification command instructs the memory controller110to schedule a precharge command to close the first row of the first bank116and to schedule an activate command to open the next row of the first bank116. In one or more implementations, the activate command is scheduled to be serviced by the memory112in parallel with execution of the second stream of PIM commands that access the second bank118. Additionally or alternatively, the precharge command is scheduled to be serviced by the memory112in parallel with execution of the second stream of PIM commands that access the second bank118.

By issuing the precharge command to close a currently open row in the first bank116while the PIM component executes PIM commands122that access the second bank118, the row precharge overhead time for the first bank116is hidden by PIM computation time. Moreover, by issuing the activate command to open a next row in the first bank116that is subsequently accessed by the PIM component120in parallel with execution of PIM commands122that access the second bank118, the row activate overhead time for the first bank116is hidden by PIM computation time. Therefore, the described techniques significantly increase PIM throughput and computational efficiency, as compared to conventional PIM-enabled systems.

FIG.2depicts a non-limiting example200in which a memory controller schedules a precharge command and an activate command to enable bank-level parallelism in accordance with the described techniques. Example200includes the PIM commands122, the memory controller110, the memory112, as well as the first bank116, the second bank118, and the PIM component120included in one or more blocks114of the memory112. Notably, the PIM commands122in the example200include a PIM command202that accesses a first row of the first bank116, an identification command204that identifies a second row of the first bank116(e.g., that is next accessed by the PIM component120), a stream of PIM commands206,208,210,212that access a first row of the second bank118, and a PIM command214that accesses a second row of the first bank116. Based on the identification command204, the memory controller110is instructed to schedule a precharge command216to close the first row of the first bank116and an activate command to open the second row of the first bank116in parallel with execution by the PIM component120of the PIM commands206,208,210,212that access the first row of the second bank118.

In accordance with the described techniques, the PIM commands122are scheduled in streams of commands which alternate between access to the first bank116and the second bank118. For example, the PIM commands122are scheduled in the following order: (1) a first stream of PIM commands122that access the first bank116, (2) a second stream of PIM commands122that access the second bank118, (3) a third stream of PIM commands122that access the first bank116, and so on. Although this functionality is described as implemented by the compiler106, it is to be appreciated that the scheduling of the PIM commands can be performed by the compiler106or through programming using framework libraries that support the described techniques.

In one or more implementations, timing constraints associated with executing the PIM commands122, opening rows of the memory112, and closing rows of the memory112are known or otherwise accessible by the compiler106. Given this, the compiler106is configured to calculate a number of commands to include in each stream of PIM commands122based on the timing constraints associated with the PIM commands122, the row precharge overhead time, and the row activate overhead time. In particular, the compiler106includes a number of PIM commands122in each stream so that an execution time of each stream of PIM commands122is equal to or greater than an amount of time which corresponds to a combination of both the row precharge overhead time and the row activate overhead time.

In the illustrated example200, for instance, each PIM command122has a computation time of eight nanoseconds, while the DRAM overhead time (e.g., row precharge overhead time and row activate overhead time) is thirty nanoseconds. In this example, the number of PIM commands included in each stream of PIM commands122is equal to or greater than thirty nanoseconds. Therefore, the compiler106schedules four PIM commands122(e.g., having a combined execution time of thirty-two nanoseconds) in each stream of PIM commands. In this way, the execution time of each stream of PIM commands122that access the first bank116hides the row precharge overhead time and the row activate overhead time in the second bank118, and vice versa. In variations, computation times and overhead times differ from the examples mentioned above without departing from the spirit or scope of the described techniques.

In at least one variation, the PIM component120utilizes a number of registers that is equivalent to the number of PIM commands included in the alternating streams of PIM commands for each bank in the block114. In the previous example involving four PIM commands in each stream, for instance, the PIM component120utilizes four registers for PIM commands assigned to the first bank116and four registers for PIM commands assigned to the second bank118. Thus, in this example, the PIM component120includes at least eight registers. It is to be appreciated that different numbers of registers are utilized by the PIM component120depending on different numbers of banks in the block114, and different numbers of PIM commands122in the alternating streams of PIM commands.

In one or more implementations, identification commands204are inserted into the plurality of PIM commands122after each stream of PIM commands that access a particular bank. In accordance with the described techniques, the identification commands204identify the next row of the particular bank that is to be accessed by a subsequent stream of PIM commands. In the illustrated example200, for instance, the compiler106inserts the identification command204after a first stream of PIM commands (e.g., including the PIM command202) that access the first row of the first bank116. Further, the identification command204identifies the next row of the first bank116(e.g., the second row) that is to be accessed by a subsequent stream of PIM commands, e.g., including the PIM command214. Although this functionality is described as performed by the compiler106, it is to be appreciated that the inserting of the identification commands204can be implemented by the compiler106or by programming which uses framework libraries that support the described techniques.

The PIM commands122, including the identification commands204, are generated by the compiler106running on the core108of the host102. In implementations, the core108of the host102issues the PIM commands122and the identification commands204to the memory controller110with physical memory addresses. The identification command204, for example, includes a physical memory address that identifies the next row that is accessed by a bank. Notably, the identification commands204identify the next row of a respective bank without causing the PIM component120to perform operations that access the next row of the respective bank. In other words, the identification command204indicates intent to access a row in the future, but does not perform a load operation, a store operation, or other operations that actually access the identified row at the time. In at least one implementation, the identification commands204follow write command semantics since the core108does not expect any data to be returned from the memory112. As further described below, the memory controller110is configured to recognize the identification commands204and schedule DRAM commands based on the row identified by the identification commands204.

After issuing the PIM command202that accesses the first row of the first bank116in the example200, the memory controller110detects that the next command in the PIM queue is the identification command204. Upon encountering the identification command204in the PIM queue, the memory controller110inspects the physical memory address included in the identification command204to identify the second row of the first bank116. If the identification command204includes a row of the first bank116that is already open in the first bank116, then the memory controller110does not schedule any additional DRAM commands. In contrast, if the identification command204identifies a new row of the first bank116(e.g., the second row of the first bank116), then the memory controller110schedules a precharge command216to close the first row of the first bank116and an activate command218to open the second row of the first bank116.

In one or more implementations, the memory controller110schedules the precharge command216and the activate command218to close the first row of the first bank116and open the second row of the first bank116, respectively, in parallel with execution of the PIM commands206,208,210,212, which cause the PIM component120to perform operations that access the second bank118. To do so, the memory controller110schedules the precharge command216and the activate command218in an interleaved fashion. As shown in the illustrated example200, for instance, the memory controller110schedules the precharge command216, followed by a first set of PIM commands206,208which operate on data maintained in the second bank118, followed by the activate command218, followed by a second set of PIM commands210,212which also operate on data maintained in the second bank118. In at least one variation, the precharge command216is scheduled to be serviced in parallel with the first set of PIM commands206,208, while the activate command218is scheduled to be serviced in parallel with the second set of PIM commands210,212.

To implement such an interleaved scheduling policy, the memory controller110initially enqueues the precharge command216and the activate command218in a DRAM command queue, which is maintained separately from the PIM queue. In at least one implementation, the memory controller110is configured to calculate how many subsequent PIM commands that access the second bank118have a combined execution time that is equal to or greater than the row precharge overhead time. This calculation is based on timing constraints which are known or otherwise accessible by the memory controller110. In the illustrated example200, for instance, each PIM command122has a computation time of eight nanoseconds, while the row precharge overhead time is fifteen nanoseconds. Therefore, the memory controller110determines that two subsequent PIM commands (e.g., PIM commands206,208) have a combined execution time (e.g., sixteen nanoseconds) that is equal to or greater than the row precharge overhead time.

The memory controller110is configured to interleave the determined number of subsequent PIM commands that access the second bank118with the precharge command216and the activate command218. In the illustrated example200, for instance, the memory controller110is configured to schedule the precharge command216to close the first row of the first bank116immediately after execution of the PIM command202that last accesses the first row of the first bank116. Further, the memory controller110is configured to schedule the activate command218to open the second row of the first bank116immediately after execution of the determined number of commands that access the second bank118. In this way, the first row of the first bank116is closed in parallel with execution of a first set of PIM commands206,208, which operate on data maintained in the second bank118. Since each stream of PIM commands includes a sufficient amount of PIM execution time to cover the row precharge overhead time and the row activate overhead time, the second row of the first bank116is opened in parallel with execution of a second set of PIM commands210,212, which access the second bank118.

In at least one implementation, the precharge command216and the activate command218are enqueued in the DRAM command queue, one after another. In accordance with such implementations, the memory controller110is configured to schedule the precharge command216to open the first row of the first bank116immediately after the PIM command202is executed. Rather than issuing the activate command218after execution of some determined number of subsequent PIM commands which access the second bank118, as described above, the memory controller110waits to issue the activate command218until after the row precharge overhead time has elapsed for the precharge command216. In this way, the first row of the first bank116is closed in parallel with execution of a first portion of the stream of PIM commands206,208,210,212that access the second bank118. Since each stream of PIM commands includes a sufficient amount of PIM execution time to cover the row precharge overhead time and the row activate overhead time, the second row of the first bank116is opened in parallel with execution of a second portion of the stream of PIM commands206,208,210,212that access the second bank118.

In order to implement an interleaved scheduling policy, the memory controller110includes additional hardware in at least one variation. For example, the memory controller110includes additional hardware to calculate the number of subsequent PIM commands (e.g., commands206,208) having a combined execution time that is equal to or greater than the row precharge overhead time. Additionally or alternatively, the memory controller110includes additional hardware to delay issuance of the activate command218based on the row precharge overhead time.

In other variations, the memory controller110does not include such additional hardware. In accordance with the described techniques, the memory controller110schedules the precharge command216and the activate command218in a non-interleaved fashion. For example, after issuing the PIM command202that last accesses the first row of the first bank116, the memory controller110determines that the next command in the PIM queue is the identification command204that identifies the second row of the first bank116. Based on the identification command204, the memory controller110schedules the precharge command216to close the first row of the first bank116immediately after the PIM command202is executed. Without the additional hardware mentioned above, the memory controller110does not schedule the subsequent PIM commands206,208,210,212that access the second bank118until the first row of the first bank116is closed. After the row precharge overhead time has passed, the memory controller110schedules the activate command218to open the second row of the first bank116in parallel with execution of the subsequent PIM commands206,208,210,212. Accordingly, in implementations in which the memory controller110implements the non-interleaved scheduling policy, the row activate overhead time is overlapped with execution of the PIM commands206,208,210,212that access the second bank118. However, the row precharge overhead time is not overlapped with execution of the PIM commands206,208,210,212that access the second bank118.

While the memory controller110overlaps only the row activate overhead time in implementations involving the non-interleaved scheduling policy, it does so without utilizing additional hardware at the memory controller110. Thus, implementations involving the non-interleaved scheduling policy process PIM commands in a shorter amount of time, as compared to conventional techniques, and do so with reduced hardware complexity, as compared to the interleaved scheduling policy.

Although described with respect to the first bank116and the second bank118, it is to be appreciated that the memory controller110is configured to issue the PIM commands122, the precharge command216, and the activate command218to each of the plurality of blocks114in parallel. As discussed above, in one or more implementations, the memory112includes a plurality of blocks114, which each include a bank (e.g., the first bank116) assigned to a first group, and an additional bank (e.g., the second bank118) assigned to a second group. In accordance with the example200, the precharge command216causes banks assigned to the first group (e.g., including the first bank116) to open the first row in parallel. Moreover, the activate command218causes banks assigned to the first group (e.g., including the first bank116) to open the second row in parallel. Furthermore, the PIM commands206,208,210,212cause the PIM components120in each block114to perform operations that access banks assigned to the second group (e.g., including the second bank118) in parallel.

Therefore, in accordance with the interleaved scheduling policy, banks assigned to the first group close a currently open row of the respective bank and open a subsequently accessed row of the respective bank concurrently while the PIM components120execute PIM commands122that access banks assigned to the second group. Further, in accordance with the non-interleaved scheduling policy, banks assigned to the first group open a subsequently accessed row of a respective bank concurrently while the PIM components120execute PIM commands122that access banks assigned to the second group.

In one or more variations, the memory controller110is configured to schedule the activate command218to open the next row in a group of banks that is to be accessed by the PIM components120without utilizing the identification command204. In at least one such variation, for instance, the memory controller110does so by inspecting the physical memory addresses associated with PIM commands122that are enqueued in the PIM queue. As discussed above, the PIM commands122received by the memory controller110, via the core108, include physical memory addresses (e.g., row and column addresses) that are accessed by the PIM commands. Since the PIM commands122are not re-ordered in the PIM queue, the memory controller110can compare earlier enqueued PIM commands to later enqueued PIM commands to identify the first PIM command that accesses a different row in a same group of banks, as compared to an earlier enqueued PIM command. For example, the memory controller110inspects the PIM queue to determine that the PIM command214is the first command in the PIM queue that accesses the second row in the first group of banks, e.g., including the first bank116. Based on this determination, the memory controller110schedules the activate command218to proactively open the second row in the first group of banks in parallel with execution of PIM commands206,208,210,212that access the second group of banks, e.g., including the second bank118.

In at least one additional variation, the memory controller110includes a prediction unit to predict the next row accessed by a group of banks. Notably, PIM command streams are scheduled in predictable sequences. For example, in many cases, the PIM command streams are scheduled in the following order: (1) a first stream of commands that access a first row of the first group of banks, (2) a second stream of commands that access a first row of the second group of banks, (3) a third stream of commands that access a second row of the first group of banks, (4) a fourth stream of commands that access a second row of the second group of banks, and so forth. Given this, it is possible to accurately predict a next row that is to be accessed by a group of banks based on a pattern of previously issued PIM commands and a row of the group of banks with respect to which the memory controller110is currently issuing PIM commands122.

Based on the pattern of the PIM commands, for example, the memory controller110predicts that the next row accessed by a group of banks is a same row with respect to a currently-accessed row, a next sequential row in comparison to the currently-accessed row, a next sequential row in comparison to a previously-accessed row, etc. The memory controller110can, therefore, schedule the activate command218to open the row of the first group of banks that is predicted to be accessed next by the PIM components120in parallel with execution of PIM commands206,208,210,212which access the second group of banks.

In one or more implementations, a prediction policy implemented by the prediction unit is enabled or disabled and/or made more or less aggressive depending on prediction success rate. For example, the memory controller110determines whether a prediction was successful based on whether the row that is opened based on the prediction is the next row in the group of banks that is actually accessed by the PIM components120. If the prediction is successful, the prediction policy is enabled and/or optionally made more aggressive. If the prediction is unsuccessful, the prediction policy is disabled and/or made less aggressive.

In at least one implementation, the core108of the host102sends hints to the prediction unit to influence the prediction policy. In one example, the core108sends a hint to the prediction unit indicative of an amount of data that is to be processed. In this example, the prediction unit predicts a number of activate commands218to issue based on the amount of data indicated by the hint. For instance, the prediction unit predicts to issue a larger number of activate commands218to proactively open a larger number of rows based on a larger amount of data indicated by the hint. In another example, a hint is sent by the core108that indicates a number of data structures involved in the PIM commands122. In this example, the rows that the prediction unit predicts to proactively open are determined, in part, based on the number of data structures. In an example involving three data structures, for instance, the prediction unit predicts to open row (x) for a first data structure, then row (y) for a second data structure, followed by row (z) for a third data structure. In the following three activate commands, the prediction unit predicts to activate row (x+1) for the first data structure, then row (y+1) for the second data structure, followed by row (z+1) for the third data structure.

FIG.3depicts a non-limiting example300showing an improvement in computational efficiency of the described techniques over conventional techniques. The example300includes a plurality of PIM commands that are issued in the following order: (1) a first stream of PIM commands302which access a first row of banks in a first group304, (2) a second stream of PIM commands306which access a first row of banks in a second group308, and (3) a third stream of PIM commands310which access a second row of banks in the first group304. Further, a variety of DRAM commands are issued in order to execute the PIM commands302,306,310.

In a prior art example312, a conventional memory controller waits for PIM execution time to complete in one group of banks before issuing DRAM commands that are utilized to execute the PIM commands in another group of banks. For example, in order to execute the PIM commands310that access the second row of the first group304of banks, the conventional memory controller first waits for one or more PIM components to finish executing the PIM commands306that access the second row of the second group308of banks. Then, the conventional memory controller issues the precharge command314to close a previously opened row (e.g., the first row) in the first group304of banks and an activate command316to open the row (e.g., the second row) in the first group304of banks that is accessed by the PIM commands310. Notably, the row precharge overhead time and the row activate overhead time for the precharge command314and the activate command316, respectively, are not overlapped with execution of PIM commands that access a different group of banks. In other words, banks in the first group304sit idle (e.g., without servicing memory commands) while banks in the second group308service memory commands, and vice versa.

Example318illustrates an order in which the memory controller110issues the PIM commands302,306,310and the DRAM commands that are utilized to execute the PIM commands in accordance with the described techniques. Specifically, example318illustrates how the memory controller110schedules the DRAM commands in accordance with the non-interleaved scheduling policy. For example, an identification command that identifies the second row of the first group304of banks is enqueued in the PIM queue after the PIM commands302that access the first row of the first group304of banks. Based on the identification command, the memory controller110first issues the precharge command314to close the first row of the first group304of banks immediately after the PIM commands302are executed. Then, the memory controller110issues the activate command316to open the second row in the first group304of banks in parallel with execution of the PIM commands306that access the second group of banks. Notably, the DRAM commands are scheduled in a similar manner to that illustrated in example318for implementations in which the memory controller110schedules the DRAM commands based on an inspection of enqueued PIM commands and/or a prediction policy rather than the identification commands204.

Example320illustrates an order in which the memory controller110issues the PIM commands302,306,310and the DRAM commands that are utilized to execute the PIM commands in accordance with the described techniques. In particular, example320illustrates how the memory controller110schedules the DRAM commands in accordance with the interleaved scheduling policy. For example, an identification command that identifies the second row of the first group304of banks is enqueued in the PIM queue after the PIM commands302that access the first row of the first group304of banks. Based on the identification command, the memory controller110first issues the precharge command314to close the first row of the first group304of banks immediately after the PIM commands302are executed. As shown, the precharge command314is scheduled to be serviced in parallel with execution of a first portion of the PIM commands306that access the second group308of banks. After the row precharge overhead time has passed, the memory controller110issues the activate command316to open the second row in the first group304of banks in parallel with execution of a second portion of the PIM commands306that access the second group308of banks.

Thus, in example318, the memory controller110overlaps row activate overhead time in one group of banks with execution of PIM commands that access a different group of banks. Further, in example320, the memory controller110overlaps row activate overhead time and row precharge overhead time in one group of banks with execution of PIM commands that access a different group of banks. By doing so, the described techniques enable processing of the PIM commands302,306,310in a shorter amount of time, as compared to the prior art example312, leading to increased computational efficiency and improved PIM throughput.

FIG.4depicts a procedure400in an example implementation showing an order in which PIM commands are scheduled and transmitted to a memory controller in accordance with the described techniques. In the procedure400, a program is compiled to generate a plurality of commands for execution by a processing in memory component embedded in a memory, the memory including a first group of banks and a second group of banks (block402). By way of example, the compiler106compiles a program to generate a plurality of PIM commands122for execution by the PIM component(s)120embedded in the memory module104. The memory module104, for instance, includes a memory112organized into a plurality of blocks114that each include a bank (e.g., the first bank116) assigned to a first group304, a bank (e.g., the second bank118) assigned to a second group308, and a respective processing in memory component120.

A first stream of commands which cause the processing in memory component to perform operations that access the first group of banks are transmitted to a memory controller (block404). For example, the core108of the host102transmits, to the memory controller110, a first stream of the PIM commands122which cause the PIM component120to perform operations that access the first group304of banks.

An identification command is transmitted to the memory controller that identifies a next row of the first group of banks that is to be accessed by the processing in memory component (block406). For example, the core108of the host102transmits, to the memory controller110, an identification command204which identifies a next row of the first group304of banks that is to be accessed by the PIM component120. In one or more implementations, the compiler106inserts the identification command204after the first stream of PIM commands122that access the first group304of banks. In accordance with the described techniques, the identification command204instructs the memory controller110to schedule an activate command218to open the next row in the first group304of banks in parallel with execution of a subsequent stream of PIM commands that access the second group308of banks. Additionally or alternatively, the identification command204instructs the memory controller110to schedule a precharge command216to close a previously opened row in the first group304of banks in parallel with execution of a subsequent stream of PIM commands that access the second group308of banks.

A second stream of commands which cause the processing in memory component to perform operations that access the second group of banks are transmitted to the memory controller (block408). For example, the core108of the host102transmits, to the memory controller110, a second stream of PIM commands122which cause the PIM component120to perform operations that access the second group308of banks. In one or more implementations, the compiler106determines a number of PIM commands to include in the first stream of PIM commands and the second stream of PIM commands based on timing constraints associated with the PIM commands122and DRAM commands. By way of example, the compiler106determines to include a sufficient number of PIM commands122in the first stream and the second stream so that the PIM commands122in each stream have a combined execution time that is equal to or greater than the row precharge overhead time and the row activate overhead time. In this way, the PIM component120can execute the first stream of PIM commands122that access the first group304of banks for long enough to hide the overhead for closing a row and opening a new row in the second group308of banks, and vice versa.

FIG.5depicts a procedure500in an example implementation of scheduling a precharge command and an activate command to enable bank-level parallelism in accordance with the described techniques. In the procedure500, a plurality of commands are received for execution by a processing in memory component embedded in a memory, the memory including a first bank and a second bank, the plurality of commands including an identification command that identifies a next row of the first bank that is to be accessed by the processing in memory component (block502). For example, the memory controller110receives a plurality of PIM commands122for execution by the PIM component120embedded in the memory module104. In one or more implementations, the memory module104includes a memory112that is organized into a plurality of blocks114, each including a first bank116and a second bank118. The plurality of PIM commands122, for instance, include a first stream of PIM commands122which cause the PIM component120to perform operations that access a first row of the first bank116, the identification command204which identifies the next row of the first bank116that is subsequently accessed by the PIM component120, and a second stream of PIM commands122which cause the PIM component120to perform operations that access the second bank118. Notably, the identification command204identifies a next row in the first bank116that is to be accessed by the PIM component120without causing the PIM component120to perform operations (e.g., load operations, store operations) that access the next row.

A precharge command is scheduled, based on the identification command, to close a first row of the first bank in parallel with execution of a first set of the commands which cause the processing in memory component to perform operations that access the second bank (block504). For example, the identification command204instructs the memory controller110to schedule the precharge command216to close the first row of the first bank116in parallel with execution of a first set of PIM commands122that access the second bank118. To do so, the memory controller110calculates a first set of the PIM commands122(e.g., PIM commands206,208) having a combined execution time that is equal to or greater than the row precharge overhead time. Further, the memory controller110schedules the precharge command216to close the first row of the first bank116in parallel with execution of the first set of PIM commands (e.g., PIM commands206,208) that access the second bank118.

An activate command is scheduled, based on the identification command, to open the next row of the first bank in parallel with execution of a second set of the commands which cause the processing in memory component to perform operations that access the second bank (block506). For example, the identification command204instructs the memory controller110to schedule the activate command218to open the next row of the first bank116in parallel with execution of a second set of PIM commands122that access the second bank118. To do so, the memory controller110schedules an activate command to open the next row of the first bank116that is included in the identification command204immediately after the determined first set of PIM commands (e.g., PIM commands206,208) are executed. Since the second stream of PIM commands includes a sufficient amount of PIM execution time to cover both row precharge overhead time and row activate overhead time, the next row of the first bank116is opened in parallel with a second set of PIM commands (e.g., PIM commands210,212) that access the second bank118.

It should be understood that many variations are possible based on the disclosure herein. Although features and elements are described above in particular combinations, each feature or element is usable alone without the other features and elements or in various combinations with or without other features and elements.

The various functional units illustrated in the figures and/or described herein (including, where appropriate, the host102, the memory module104, the compiler106, the core108, the memory controller110, the memory112, and the PIM component120) are implemented in any of a variety of different manners such as hardware circuitry, software or firmware executing on a programmable processor, or any combination of two or more of hardware, software, and firmware. The methods provided are implemented in any of a variety of devices, such as a general-purpose computer, a processor, or a processor core. Suitable processors include, by way of example, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a graphics processing unit (GPU), a parallel accelerated processor, a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), and/or a state machine.

In one or more implementations, the methods and procedures provided herein are implemented in a computer program, software, or firmware incorporated in a non-transitory computer-readable storage medium for execution by a general-purpose computer or a processor. Examples of non-transitory computer-readable storage mediums include a read only memory (ROM), a random-access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs).

Although the systems and techniques have been described in language specific to structural features and/or methodological acts, it is to be understood that the systems and techniques defined in the appended claims are not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claimed subject matter.