Patent Description:
In a computer with SIMD architecture, each of the parallel multiple processing units, Arithmetic Logic Units (ALUs) or small CPUs, compute simultaneously with their own data - generally <NUM> or <NUM> input operands and <NUM> output result. These data are stored in memory and are accessed independently in parallel. Thus, each processing unit can have a dedicated partition of memory and dedicated access ports to the partitions of memory. In practice, many algorithms have some shared data, which can be stored in some shared memory (to save storage cost) and be broadcasted to all processing units as one of the operands.

To enable parallel access in SIMD architecture, hardware generally introduces physically separated private memory modules and shared memory modules to hold corresponding type of data. However, such memory organization has two issues.

First, because the size of each hardware memory module is fixed while different software programs have different data sizes, these modules are inefficiently utilized, resulting in the waste of physical memory space. Second, dedicated memory copy operations have to be performed when previously considered "private" data becomes "shared" data in a later phase of the program. This causes extra power consumption and a drop in performance of the processing unit.

<CIT> relates to a technique for addressing data in a hierarchical graphics processing unit cluster. A hierarchical address is constructed based on the location of a storage circuit where a target unit of data resides. The hierarchical address comprises a level field indicating a hierarchical level for the unit of data and a node identifier that indicates which GPU within the GPU cluster currently stores the unit of data. The hierarchical address may further comprise one or more identifiers that indicate which storage circuit in a particular hierarchical level currently stores the unit of data. The hierarchical address is constructed and interpreted based on the level field.

<CIT> relates to a technique for unifying the addressing of multiple distinct parallel memory spaces into a single address space for a thread. A unified memory space address is converted into an address that accesses one of the parallel memory spaces for that thread. A single type of load or store instruction is used that specifies the unified memory space address for a thread instead of using a different type of load or store instruction to access each of the distinct parallel memory spaces.

[<CIT> discloses a plurality of processing elements (PEs) including memory local to at least one of the processing elements in a data packet-switched network interconnecting the processing elements and the memory to enable any of the PEs to access the memory. The network consists of nodes arranged linearly or in a grid to connect the PEs and their local memories to a common controller. The processor performs memory accesses on data stored in the memory in response to control signals sent by the controller to the memory. The local memories share the same memory map or space. The packet-switched network supports multiple concurrent transfers between PEs and memory.

<CIT> discloses local addressing for a processing element array by partitioning a register file memory (e.g., data columns, data rows), and adding a select column or row to be associated with each block. The select column or row allows each processing element to read data from or to write data to a different register file address. Global addressing may also be implemented by reading data from or writing data to the same register file address for each processing element.

<CIT> discloses a storage device and method for performing convolution operations. One embodiment of an apparatus to perform convolution operations comprises a plurality of processing units to execute convolution operations on input data and partial results; a unified scratchpad memory comprising a plurality of memory banks communicatively coupled to the plurality of processing units through a plurality of read/write ports, each of the plurality of memory banks partitioned to store both the input data and partial results; a control unit to allocate the input data and partial results to the memory banks to ensure a minimum quality of service in accordance with the specified number of read/write ports and the specified convolution operation to be performed.

<CIT> discloses a technique in which a shared memory controller receives, from a computing node, a request associated with a memory transaction involving a particular line in a memory pool. The request includes a node address according to an address map of the computing node. An address translation structure is used to translate the first address into a corresponding second address according to a global address map for the memory pool, and the shared memory controller determines that a particular one of a plurality of shared memory controllers is associated with the second address in the global address map and causes the particular shared memory controller to handle the request.

<CIT> discloses a hierarchical complexity for coherence protocols associated with clustered cache architectures which can be encapsulated in a simple function, i.e., that of determining when a data block is shared entirely within a cluster (i.e., a sub-tree of the hierarchy) and is private from the outside.

<CIT> discloses a data processing system which includes an instruction decoder which decodes protected memory access instructions (LDR/STR) and less-protected memory access instructions (LDNPR/STNPR) to generate control signals for controlling a load store unit. The less-protected memory access instructions are associated with less restrictive access conditions than the protected memory access instructions.

According to aspects of the present invention, there are provided methods for accessing data in a unified storage medium, a computer program, and a unified storage medium as set forth in the appended claims.

Embodiments of this disclosure provide a unified memory apparatus. The unified memory apparatus can include a unified storage medium including a first storage module having a first plurality of storage cells configured to store data, the first plurality of storage cells identified by a unique cell identifier, and a second storage module having a second plurality of storage cells configured to store data, the second plurality of storage cells identified by a unique cell identifier. The unified memory architecture can also include a processing unit in communication with the unified storage medium. The processing unit can be configured to receive a first input data from one of the first plurality of storage cells, receive a second input data from one of the second plurality of storage cells, and generate an output data based on the first and second input data.

Some embodiments of this disclosure provide a unified storage medium. The unified storage medium can include a first storage module having a first plurality of storage cells configured to store data, the first plurality of storage cells identified by a unique cell identifier, and a second storage module having a second plurality of storage cells configured to store data, the second plurality of storage cells identified by a unique cell identifier.

Some embodiments of this disclosure provide a method for organizing data in a unified memory apparatus having a unified storage medium and one or more processing units. The method can include configuring a first storage module of the unified storage medium to communicate with the one or more processing units and to include a first plurality of storage cells that are configured to store data, the first plurality of storage cells identified by a unique cell identifier. The method can also include configuring a second storage module of the unified storage medium to communicate with the one or more processing units and to include a second plurality of storage cells that are configured to store data, the second plurality of storage cells identified by a unique cell identifier. The method further includes configuring a processing unit of the one or more processing units to receive a first input data from one of the first plurality of storage cells, receive a second input data from one of the second plurality of storage cells, and generate an output data based on the first and second input data.

Some embodiments of this disclosure provide a method for organizing data in a unified storage medium having a first storage module and a second storage module. The method can include configuring the first storage module of the unified storage medium to communicate with one or more processing units and to include a first plurality of storage cells that are configured to store data, the first plurality of storage cells identified by a unique cell identifier, and configuring the second storage module of the unified storage medium to communicate with one or more processing units and to include a second plurality of storage cells that are configured to store data, the second plurality of storage cells identified by a unique cell identifier.

The unique cell identifier of the first and second plurality of storage cells can comprise a bit address including a first plurality of bits and a second plurality of bits. The first plurality of bits can indicate a target storage module of the first and second storage modules, and the second plurality of bits can indicate a target storage cell of the first and second plurality of storage cells within the target storage module. The second plurality of bits can further indicate a characteristic associated with the target storage cell, the characteristic of the target storage cell being one of private or shared. In some embodiments, the first and second storage modules are configured to communicate with a corresponding processing unit. The processing unit is configured to receive the first input data from a private storage cell, and the second input data from a shared storage cell. The unified storage medium and the processing unit are configured to be uniformly addressed by a software code or a software program. The unified storage medium is further configured to receive instructions from a compiler, the instructions including a characteristic associated with the data, wherein the characteristic associated with the data is one of private or shared. The private storage cell is configured to store private data and the shared storage cell is configured to store shared data that can be shared across the multiple processing units.

The implementations set forth in the following description of exemplary embodiments do not represent all implementations consistent with the invention. Instead, they are merely examples of apparatuses and methods consistent with aspects related to the invention as recited in the appended claims.

The disclosed embodiments provide systems and methods for organizing data stored in a unified memory architecture and accessing the target data thereof. The disclosed embodiments can resolve the aforementioned issues of conventional SIMD architecture by organizing the physical private and shared memory in a unified way. The disclosed embodiments maintain a single module of physical memory for logical private and shared memory, and can switch the view of "private" or "shared" through the accessing instructions while keeping the data itself in its original location in the physical memory.

<FIG> illustrates an exemplary neural network processing unit (NPU) architecture <NUM>. NPU architecture <NUM> can include an on-chip communication system <NUM>, an off-chip memory <NUM>, a memory controller <NUM>, a direct memory access (DMA) unit <NUM>, a Joint Test Action Group (JTAG)/Test Access End (TAP) controller <NUM>, a peripheral component interconnect express (PCIe) interface <NUM>, inter-chip links <NUM>, and the like. It is appreciated that on-chip communication system <NUM> can perform algorithmic operations based on communicated data.

On-chip communication system <NUM> can include a global manager <NUM> and a plurality of tiles <NUM>. Global manager <NUM> can include one or more cluster managers <NUM> configured to coordinate with one or more tiles <NUM>. Each cluster manager <NUM> can be associated with an array of tiles <NUM> that provide synapse/neuron circuitry for the neural network. For example, the top layer of tiles of <FIG> may provide circuitry representing an input layer to neural network, while the second layer of tiles may provide circuitry representing a hidden layer of the neural network. As shown in <FIG>, global manager <NUM> can include two cluster managers <NUM> configured to coordinate with two arrays of tiles <NUM>. Tiles <NUM> can include one or more multipliers, adders, multiply-accumulators (e.g., a set of multiply-accumulators of a SIMD architecture) and corresponding memory and can be configured to perform an operation (e.g., one or more algorithmic calculations) on the communicated data under the control of global manager <NUM>.

Off-chip memory <NUM> can include read-only memory (ROM), erasable programmable read-only memory (EPROM) or the like. Off-chip memory <NUM> can be configured to store a large amount of data with slower access speed, compared to the on-chip memory integrated within one or more processor.

Memory controller <NUM> can read, write, or refresh one or more memory devices. The memory devices can include on-chip memory and off-chip memory <NUM>. For example, the memory device can be implemented as any type of volatile or non-volatile memory devices, or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, or a magnetic or optical disk.

DMA unit <NUM> can generate memory addresses and initiate memory read or write cycles. DMA unit <NUM> can contain several hardware registers that can be written and read by the one or more processors. The registers can include a memory address register, a byte-count register, and one or more control registers. These registers can specify some combination of the source, the destination, the direction of the transfer (reading from the input/output (I/O) device or writing to the I/O device), the size of the transfer unit, and/or the number of bytes to transfer in one burst.

JTAG/TAP controller <NUM> can specify a dedicated debug port implementing a serial communications interface (e.g., a JTAG interface) for low-overhead access without requiring direct external access to the system address and data buses. The JTAG/TAP controller <NUM> can also specify an on-chip test access interface (e.g., a TAP interface) that implements a protocol to access a set of test registers that present chip logic levels and device capabilities of various parts.

Peripheral interface <NUM> can support full-duplex communication between any two endpoints, with no inherent limitation on concurrent access across multiple endpoints.

Inter-chip links <NUM> can connect all the internal components of NPU architecture <NUM>, such as on-chip communication system <NUM>, off-chip memory <NUM>, memory controller <NUM>, DMA unit <NUM>, JTAG/TAP controller <NUM>, and PCIe interface <NUM> to each other.

While NPU architecture <NUM> incorporates the embodiments of the present disclosure, it is appreciated that the disclosed embodiments can be applied to chips with SIMD architecture for accelerating some applications such as deep learning. Such chips can be, for example, GPU, CPU with vector processing ability, or neural network accelerators for deep learning. SIMD or vector architecture is commonly used to support computing devices with data parallelism, such as graphics processing and deep learning. The SIMD architecture can include multiple processing elements, wherein each of the processing elements can perform the same operation on multiple data points simultaneously.

For example, the private memory can be memory dedicated to serving data for each single processing element among multiple parallel processing elements, while shared memory can refer to memory dedicated to serving data for all parallel processing elements.

<FIG> illustrates an exemplary functionality of a layer <NUM> of neural network, including a software algorithm <NUM> and hardware <NUM>. Hardware <NUM> can include a private memory module <NUM>, a processing unit array <NUM>, a shared memory module <NUM>, a write buffer <NUM>, input operands <NUM>, output operand <NUM>, and the like. In some embodiments, hardware <NUM> can be located in a tile (e.g., tile <NUM> of <FIG>).

In some embodiments, a processing unit of processing unit array <NUM> can be an Arithmetic Logic Unit (ALU), a Floating Point Unit (FPU), a CPU, a GPU, or the like. An ALU is a fundamental building block of a computing circuit, including the CPU of computers. A single CPU can contain one or more ALUs. Generally, an ALU is a combinational digital electronic circuit that performs arithmetic and bitwise operations on integer binary numbers. Processing unit array <NUM> can include multiple processing units <NUM>, <NUM>, <NUM>, and <NUM>, for example, an array of processing units, as illustrated in <FIG>.

Private memory module <NUM> can be partitioned into separate private memory blocks, such that, each of the multiple processing units <NUM>, <NUM>, <NUM>, and <NUM> has a corresponding private memory block <NUM>, <NUM>, <NUM>, and <NUM>, as shown in <FIG>.

Input operands <NUM> can be the input data operated on by processing unit array <NUM>. In some embodiments, input operands <NUM> of <FIG> can include one or more private input operand(s) <NUM> and one or more shared input operand(s) <NUM>, as shown in <FIG>. Private input operand <NUM> can be stored in private memory module <NUM> and shared input operand <NUM> can be stored in shared memory module <NUM>.

In the application of neural networks, software algorithms <NUM> have shared data that can be stored in shared memory module <NUM> and can be broadcasted to each of the multiple processing units <NUM>, <NUM>, <NUM>, and <NUM> of processing unit array <NUM> as a shared operand <NUM>. For example, the algorithm illustrated in <FIG> is computing a vector operation of: <MAT> which is a representative operation in layer <NUM> of a neural network called out often in deep learning algorithms. With reference to equation <NUM>, "b" can include a constant value, "X" can include a shared input operand <NUM>, and "W<NUM>" can include a private input operand <NUM>.

With reference to <FIG>, the vector size can be set as any natural number. Here, a vector size of <NUM> is taken as an example, and a <NUM>-way SIMD hardware to compute the vector is used. The processing units <NUM>, <NUM>, <NUM>, and <NUM> can compute, in parallel, the following operations: <MAT> <MAT> <MAT> <MAT>.

The shaded blocks and dotted lines in <FIG> indicate how "a1" is calculated. From this calculation, it is appreciated that data in each column of the "W1" array is local to a corresponding processing unit of processing unit array <NUM> and the data can accordingly be stored in corresponding memory block of private memory module <NUM>, as a private input operand <NUM>. For example, the data in each of the first, second, third, and fourth columns of the W1 array can be stored in their corresponding memory blocks <NUM>, <NUM>, <NUM>, and <NUM> of private memory module <NUM> as private input operands.

With reference to <FIG>, the W1 array can include a matrix of stored data, wherein each element of the matrix is represented as W1ij or W1_ij (as shown later), where "i" represents the row number and "j" represents the column number in the matrix. For example, in Eq. <NUM>, W1<NUM> represents the data stored in the element located at row <NUM> and column <NUM> of the W1 array. Other commonly known notations to address elements in a matrix can be used as well.

Simultaneously, data in the X-array is utilized by all processing units <NUM>, <NUM>, <NUM>, and <NUM>, and is accordingly stored in shared memory module <NUM>, as shared input operand <NUM> and broadcasted to all components reading from shared memory module <NUM>. Equations <NUM>-<NUM> represent exemplary operations performed in layer <NUM> of a neural network processor, designed to calculate a1, a2, a3 and a4.

In some embodiments, machine learning or deep learning includes training the neural network processor to generate an end result based on input data, accomplished by implementing algorithms for one or more layers of neural processing. For example, layer <NUM> of <FIG>, represents a first layer including an algorithm configured to perform an operation using a bias b, data stored in the X array, and data stored in W1 array. A second and third layer (not shown) can include algorithms using the bias b, data stored in the X array, and data stored in W2 and W3 array. Each layer can include a different value of bias b and different parameters stored in "W" array.

With reference to <FIG>, for example, array X can include an individual's scores in different classes. The value of x1 of the array X can be student A's Math score, x2 can be the English score, x3 can be the History score, and x4 can be the Science score. The end result can be whether the individual will be granted admission in a school or rejected, based on the scores (input data). As shown in <FIG>, and described in Equations <NUM>-<NUM>, data x1-x4 is "shared" and common in calculating a1-a4.

<FIG> illustrates data sharing in multi-layer networks. Data sharing, as described herein, refers to how previously private data can become shared data in a later phase of a program. In some embodiments, neural network architecture <NUM> includes multiple layers, for example, layers <NUM> and <NUM>. In some embodiments, output operand <NUM> of layer <NUM> can be used as an input operand <NUM> for layer <NUM>. In some embodiments, the output operand <NUM> of one layer can be utilized as input operand <NUM> by one or more layers.

For example, in layer <NUM>, a1 is calculated by processing unit <NUM> of private memory module <NUM>. The data in a1 becomes a broadcasting input for layer <NUM>. Generally, a neural network can be organized in layers. Each layer can perform one or more calculations on its inputs and generate an output. The output of a layer can be passed onto a next layer for further processing. For example, an output of a previous layer can be an input for the next layer. Accordingly, the locally generated "a"s have to be either stored back to shared memory <NUM>, or stored to private memory <NUM> and copied later to shared memory <NUM>.

As an alternative solution to storing in private memory <NUM> and copying to shared memory <NUM> later, output operand <NUM> from a1 can be stored back directly to shared memory <NUM> than memory copying. Nevertheless, this alternative solution could still slow down the program. Since a single processing unit, for example processing unit <NUM>, can finish only one multiply-add operation per cycle, say Xi*W1_ij, each calculation of "a" can be performed over multiple cycles. For this reason, only one operand of W1_ij is read out from private memory <NUM> in each cycle, thus only one "X" is needed from shared memory <NUM>. Consequently, a common design of each memory module is single-read/single-write per cycle. When all "a"s are generated simultaneously by multiple processing units in the last cycle, shared memory <NUM> may not have the ability to write them all back.

In some embodiments, a write buffer <NUM> is introduced to allow shared memory <NUM> more time to consume these output operands <NUM> individually. However, when the output speed of processing unit array <NUM> is faster than the width of write buffer <NUM>, e.g., the size of A is greater than X, write buffer <NUM> may propagate a back pressure, forcing the processing unit array <NUM> to slow down, resulting in the slowdown of the overall program execution.

<FIG> illustrates a schematic diagram of an exemplary hardware system <NUM> including unified organization of memory modules. Hardware system <NUM> includes a unified storage medium <NUM> and processing units <NUM>, <NUM>, <NUM>, and <NUM>. Unified storage medium <NUM> includes one or more storage modules <NUM>, each including storage cells <NUM> configured to store input operand <NUM>, output data <NUM>. Multiple storage modules <NUM> can be merged into a single medium to form unified storage medium <NUM>. Each storage module <NUM> can include a private storage module <NUM> and a shared storage module <NUM>.

Hardware system <NUM> can include multiple processing units <NUM>, <NUM>, <NUM>, and <NUM>. Each of the multiple processing units of the processing unit array <NUM> is configured to communicate with one or more storage modules. For example, processing unit <NUM> can receive private input operand <NUM> from private storage module <NUM>. Processing unit <NUM> can also receive shared input operand <NUM> from one or more shared storage modules <NUM>. In some embodiments, processing unit array <NUM> is configured to receive private input operand <NUM> from private storage module <NUM>, receive shared input operand <NUM> from shared storage module <NUM>, and generate an output operand <NUM> based on private input operand <NUM> and shared input operand <NUM>.

As illustrated in <FIG>, each of the storage cells <NUM> can be uniquely identified by a unique identifier <NUM>. Unique identifier <NUM> can be a bit address including high-order bits <NUM> and low-order bits <NUM>, or a byte address including high-order and low-order bytes, or a combination thereof. In computing, high-order bits <NUM> can be the most significant bit (MSB). The MSB can also be referred to as the left-most bit due to the convention in positional notation of writing more significant digits further to the left. Low-order bits <NUM>, on the other hand, are referred to as bits in the right-most position. For example, in a unique identifier <NUM> having a bit address "2_E5", the high-order bits <NUM> refer to the left-most bit, i.e. "<NUM>" and the low-order bits <NUM> refer to the bits on the right side, i.e. "E5".

In some embodiments, storage cell <NUM> is a private storage cell <NUM> or a shared storage cell <NUM>. Private storage cells <NUM> can be located within private storage module <NUM>. Shared storage cells <NUM> can be located within shared storage module <NUM>. High-order bits <NUM> of unique identifier <NUM> are configured to indicate a target storage module for operand (<NUM>, <NUM>) and low-order bits <NUM> of unique identifier <NUM> are configured to indicate a target storage cell within target storage module, for operand (<NUM>, <NUM>). For example, unique identifier <NUM> having a bit address "2_E5" refers to storage module "<NUM>", and storage cell "E5" within storage module "<NUM>". In other words, high-order bits <NUM> can also indicate the processing unit to which the storage module is "private" to, and low-order bits <NUM> indicate the location within the storage module.

It is to be appreciated that private storage cells <NUM> and shared storage cells <NUM> are physically indistinguishable storage cells and are not pre-labelled as such. The attribute of "private" and "shared" for a storage cell is determined based on the compiler-generated instructions programmed to address the data. For example, data can be stored in any cell. During a read step, if the compiler-generated instructions refer to the data as "private," the data may be read out in parallel as private input operand <NUM>. Alternatively, if the compiler-generated instructions refer to the data as "shared," the data may be read out as shared input operand <NUM>.

In some embodiments, unique identifier <NUM> includes other characters, for example, numeric characters, alpha-numeric characters, hexadecimal numerals (e.g., shown in <FIG>), octal numerals, or the like, addressable by a software addressing mode.

Referring back to <FIG>, processing unit array <NUM> or each of the multiple processing units can generate output data <NUM>. Output data <NUM> can be a private output data <NUM> or a shared output data <NUM>, determined by the operations in the next layer of a multi-layered algorithm for a neural network processor. As illustrated in <FIG>, output data <NUM> can be considered private output data <NUM> since it is written back to unified storage medium in parallel in each of the storage modules <NUM>.

In some embodiments, neural network processors comprise a compiler (not shown). The compiler is a program or computer software that transforms computer code written in one programming language into another programming language to create an executable program. In machining applications, a compiler can perform a variety of operations, for example, pre-processing, lexical analysis, parsing, semantic analysis, conversion of input programs to an intermediate representation, code optimization, and code generation, or combinations thereof.

<FIG> is a process flowchart of an exemplary data organization operation <NUM>, consistent with embodiments of the present disclosure. For example, data organization operation <NUM> can be performed by an on-chip communication system (e.g., on-chip communication system <NUM>).

Step <NUM> includes configuring a storage module (e.g., storage module <NUM>) of a unified storage medium (e.g., unified storage medium <NUM>) to include multiple storage cells (e.g. storage cells <NUM>). In some embodiments, step <NUM> includes configuring a private storage module (e.g., private storage module <NUM>) to include private storage cells (e.g., private storage cell <NUM>) and/or a shared storage module <NUM> (e.g., shared storage module <NUM>) to include shared storage cells (e.g., shared storage cell <NUM>). Configuring a storage module to include storage cells can comprise allocating storage space based on the total storage space available, software programs or algorithms, hardware limitations, time restrictions, and the like. If a software application or an algorithm is multi-layered and requires multiple layers of computation including more shared data than private data, the storage module can be configured to comprise more shared storage cells or more shared storage modules.

Step <NUM> includes configuring a storage medium (e.g., unified storage medium <NUM> of <FIG>) to communicate with a processing unit (e.g., processing unit array <NUM>) or multiple processing units. In some embodiments, the processing unit is an Arithmetic Logic Unit (ALU), a Floating Point Unit (FPU), a Central Processing Unit (CPU), or a Graphics Processing Unit (GPU). A single CPU can contain one or more ALUs. Generally, an ALU is a combinational digital electronic circuit that performs arithmetic and bitwise operations on integer binary numbers. The processing unit can include multiple processing units, for example, an array of processing units configured to operate in parallel.

Communicating with a processing unit can include receiving data generated by the processing unit, or providing stored data to the processing unit. The storage medium can be the source of data to be computed on or the target of data storage. In some embodiments, the hardware system comprises a single processing unit configured to receive data from multiple storage modules. The hardware system can also include a unique processing unit for each storage module, configured to receive data only from the corresponding storage module.

Step <NUM>, processing unit (e.g., processing unit array <NUM>) generates output data (e.g., output data <NUM>) based on the instructions generated by a compiler. In some embodiments, the compiler may be a program or computer software that transforms computer code written in one programming language into another programming language to create an executable program. The compiler can generate a set of instructions configured to access data from a storage medium, execute a desired operation on the accessed data, generate output data based on the operation, and store the generated output data back into the storage medium for subsequent processing. The instructions can also include assigning a characteristic to the input and the output data. The characteristic of the data can be private, shared, restricted, or the like.

In the example discussed here, compiler generates the following code for the vector operation "A=X*W1", where "X" can be considered as operand <NUM>, and "W1" can be considered as operand <NUM>. The set of instructions will be described with reference to <FIG>, in accordance with embodiments of the disclosure.

The instructions in the aforementioned set of instructions generally comprise an operation on the data, characteristic of the data, and a target location within the storage medium.

In some embodiments, operation on the data includes load (reading), store (writing), arithmetic operations, (e.g., addition, subtraction, multiplication, division) copy, paste, and the like. Characteristic of the data can refer generally to the accessibility of the data within the storage medium. Characteristic of the data can include private, shared, restricted, allowed, global, local, or combinations thereof. Data, in general, is referred to as an operand. Data can be an input operand, for example, operand <NUM> (OP1) and operand <NUM> (OP2), or an output data based on the vector operation being performed.

In the set of instructions i1- i4, the subfield of load/store instructions implies how to load/store the data. Subfield ". SHARED" implies that the data should be read or written as shared data. In this mode, both high-order bits (e.g., <NUM> of <FIG>) and low-order bits (e.g., <NUM> of <FIG>) are utilized to determine the target location of input operand or output data. Subfield ". SIMD" implies that the data should be read or written as private data in parallel, wherein, the high-order bits can be disregarded by hardware and the low-order bits are utilized to determine the target location of input operand or output data.

In instruction i1, each processing unit (e.g., <NUM>, <NUM>, <NUM>, and <NUM> of <FIG>) reads input operand <NUM> (e.g., private input operand <NUM>) in parallel. The high-order bit "<NUM>" in bit address "0_00" is not utilized, and the low-order bits "<NUM>" indicate the storage cell and a characteristic of the storage cell. (e.g., private storage cell <NUM>) For example, with reference to <FIG>, all data in row <NUM> of the "W1" array (W1_1i) is read out simultaneously but separately to each corresponding processing unit. SIMD" field implies that the data should be read in parallel.

In instruction i2, input operand <NUM> (e.g., shared input operand <NUM>) is read once and broadcast to all processing units, as illustrated in <FIG>. The high-order bit "<NUM>" in bit address "0_F0" indicates the storage module where the data is stored, and the low-order bits "F0" indicate the storage cell and a characteristic of the storage cell in which the data is stored (e.g., shared storage cell <NUM>). For example, with reference to <FIG>, the data in "X1" of the "X" array is read out read once and broadcast to each corresponding processing unit. SHARED field implies that the data should be read as shared data between all processing units.

In instruction i3, processing unit performs multiplication of input operands <NUM> and <NUM>, as defined by the vector operation, to generate an output data "A". The arithmetic operation can include basic arithmetic functions of addition, subtraction, multiplication, or division, or combinations thereof. In some embodiments, processing unit is configured to perform complex arithmetic and algebraic functions, logarithmic functions, exponentiation, or the like.

In instruction i4, generated output data "A" in instruction i3 is stored in parallel back to storage medium for further processing. Generated output data "A" (e.g., output data <NUM>) can be used as the input operand in the next layer of the multi-layered algorithm. The high-order bit "<NUM>" in bit address "0_F1" is not utilized by hardware, and the low-order bits "F1" indicate the storage cell and a characteristic of the storage cell (e.g., shared storage cell <NUM>) for the output data to be stored. For example, with reference to <FIG>, output data <NUM> may be temporarily stored in a temporary storage (e.g., write buffer <NUM>) before storing it in the shared or private storage module of the unified storage medium.

In step <NUM>, generated output data is stored back in the unified storage medium for further processing. Generally, a neural network can be organized in multiple layers. The output of a layer can be passed onto a next layer for further processing. For example, an output of a previous layer can be an input for the next layer.

Claim 1:
A unified memory apparatus (<NUM>) for a neural multi-layer network (<NUM>), the apparatus comprising:
a unified storage medium (<NUM>) comprising a plurality of storage modules (<NUM>) of physical memory in the unified storage medium, each storage module having a plurality of storage cells (<NUM>) configured to store data, wherein each of the plurality of storage cells is identifiable by a unique cell identifier (<NUM>),
wherein the unified storage medium (<NUM>) is configured to receive compiler generated instructions from a compiler, the compiler generated instructions including a characteristic associated with the data, the characteristic being one of private or shared; and
a plurality of processing units (<NUM>, <NUM>, <NUM>, <NUM>) each configured to generate an output operand (<NUM>) based on data read from the unified storage medium, wherein:
each of the plurality of storage modules (<NUM>) is maintained for logical private and shared memory and is addressable as a private storage module and as a shared storage module, based on the characteristic included in the compiler generated instructions programmed to address the data, while the data is kept in its original location in the physical memory,
during a read step performed by the processing units (<NUM>, <NUM>, <NUM>, <NUM>), if the compiler generated instructions refer to the data as private, the data is read out in parallel as a private input operand (<NUM>) and if the compiler-generated instructions refer to the data as shared, the data is read out as a shared input operand (<NUM>),
each of the storage cells (<NUM>, <NUM>) is configured to store input operands, and output operands, and
an output operand (<NUM>) from a layer (<NUM>) of the multi-layer network is used as an input operand (<NUM>) for another layer (<NUM>) of the multi-layer network.