Intelligence processing unit and its tensor concatenation method

An intelligence processing unit is coupled to an external memory and includes a memory, a direct memory access (DMA) circuit, and a vector accelerator. The external memory stores a first tensor and a second tensor. The DMA circuit performs the following steps: reading a first part of the first tensor from the external memory; storing the first part of the first tensor in the memory; reading a second part of the second tensor from the external memory; and storing the second part of the second tensor in the memory. The vector accelerator includes a register circuit and performs the following steps: storing P bytes of the first part of the first tensor in a target row of the register circuit; storing Q bytes of the second part of the second tensor in the target row of the register circuit; and writing data of the target row into the memory.

This application claims the benefit of China application Serial No. CN202310602205.6, filed on May 25, 2023, the subject matter of which is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to tensor operations of artificial intelligence (AI), and, more particularly, to tensor concatenation methods and intelligence processing units (IPUs).

2. Description of Related Art

FIG.1shows a schematic diagram of conventional tensor concatenation. The conventional tensor concatenation method concatenates tensors by moving multiple tensors between the memory110and the memory120. More specifically, as shown inFIG.1, the memory110stores4tensors to be concatenated (each being a3-dimensional tensor): the tensor111(of shape [100,32,4]), the tensor112(of shape [100,32,4]), the tensor113(of shape [100,32,1]), and the tensor114(of shape [100,32,1]). By writing the tensors into the memory120and reading them from the memory120, the concatenated tensor115(of shape [100,32,10]) can be obtained.

Note that in the example ofFIG.1, the axis of the concatenation operation is the innermost dimension of the tensors, and except for the innermost dimension, all other dimensions of the tensors to be concatenated are the same (i.e., [100,32,x], x=1 for the tensor111and the tensor112, and x=4 for the tensor113and the tensor114).

The disadvantage of the conventional tensor concatenation is that when the data transfer between the memory110and the memory120is performed by a direct memory access (DMA) circuit, it costs the DMA circuit a lot of time to arrange data due to the fact that the data must be read from consecutive memory addresses in the same read operation, and the data must be written to consecutive memory addresses in the same write operation. For example, assuming that the DMA circuit reads 32 bytes of the tensor111from the memory110in a read operation, when writing the 32 bytes of data to the memory120, the DMA circuit must perform8(=32/4, where 4 is the innermost dimension of the tensor111, corresponding to columns1to4of the memory120) write operations, each writing 4 bytes (because the 4 bytes in each row are consecutive addresses). Note that columns 1 to 4 of the kth row and columns1to4of the (k+1)throw are not consecutive addresses, k being a positive integer.

Continuing the previous paragraph, similarly, the DMA circuit needs to perform 8 (=32/4, where 4 is the innermost dimension of the tensor112, corresponding to columns5to8of the memory120),32(=32/1, where 1 is the innermost dimension of the tensor113, corresponding to column9of the memory120), and32(32 32/1, where 1 is the innermost dimension of the tensor114, corresponding to column10of the memory120) write operations when writing consecutive32bytes of the tensor112, the tensor113, and the tensor114to the memory120respectively. Therefore, the DMA circuit needs to perform 100*32*4/4=3200 times, 100*32*4/4=3200 times, 100*32*1/1=3200 times, and 100*32*1/1=3200 times of write operations to write the tensor111, the tensor112, the tensor113, and the tensor114to the memory120respectively (a total of 3200*4=12800 write operations are required). Such low tensor concatenation efficiency affects the performance of electronic devices, resulting in poor user experience.

SUMMARY OF THE INVENTION

According to one aspect of the present invention, an intelligence processing unit (IPU) is provided. The IPU is coupled to an external memory storing a first tensor and a second tensor. The IPU includes a memory, a direct memory access (DMA) circuit, and a vector accelerator. The DMA circuit is coupled to the external memory and the memory and configured to perform the following steps: reading a first part of the first tensor from the external memory; storing the first part of the first tensor in the memory; reading a second part of the second tensor from the external memory; and storing the second part of the second tensor in the memory. The vector accelerator includes a register circuit. The vector is coupled to the memory and configured to perform the following steps: storing P bytes of the first part of the first tensor in a target row of the register circuit, P being a positive integer; storing Q bytes of the second part of the second tensor in the target row of the register circuit, Q being a positive integer; and writing data of the target row into the memory.

According to another aspect of the present invention, a tensor concatenation method is provided. The tensor concatenation method is implemented in an IPU. The IPU is coupled to an external memory and includes a memory and a register circuit. The external memory stores a first tensor and a second tensor. The tensor concatenation method includes the following steps: reading a first part of the first tensor from the external memory; storing the first part of the first tensor in the memory; reading a second part of the second tensor from the external memory; and storing the second part of the second tensor in the memory; and storing P bytes of the first part of the first tensor in a target row of the register circuit, P being a positive integer; storing Q bytes of the second part of the second tensor in the target row of the register circuit, Q being a positive integer; and writing data of the target row into the memory.

The technical means embodied in the embodiments of the present invention can solve at least one of the problems of the prior art. Therefore, compared to the prior art, the present invention can improve the efficiency of tensor concatenation.

These and other objectives of the present invention no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiments with reference to the various figures and drawings.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The following description is written by referring to terms of this technical field. If any term is defined in this specification, such term should be interpreted accordingly. In addition, the connection between objects or events in the below-described embodiments can be direct or indirect provided that these embodiments are practicable under such connection. Said “indirect” means that an intermediate object or a physical space exists between the objects, or an intermediate event or a time interval exists between the events.

The disclosure herein includes an intelligence processing unit (IPU) and its tensor concatenation method. On account of that some or all elements of the IPU could be known, the detail of such elements is omitted provided that such detail has little to do with the features of this disclosure, and that this omission nowhere dissatisfies the specification and enablement requirements. Some or all of the processes of the tensor concatenation method may be implemented by software and/or firmware and can be performed by the IPU. A person having ordinary skill in the art can choose components or steps equivalent to those described in this specification to carry out the present invention, which means that the scope of this invention is not limited to the embodiments in the specification.

FIG.2is a functional block diagram of an electronic device200according to an embodiment of the present invention. The electronic device200includes an external memory210and an IPU220. The IPU220is coupled to the external memory210and includes a convolution engine221, a vector engine222, a direct memory access (DMA) circuit223, a memory224, and a vector accelerator226. The convolution engine221and the vector engine222can respectively perform convolution operations and vector operations on tensors. The vector accelerator226includes a register circuit228. The IPU220accesses the external memory210through the DMA circuit223. The external memory210, the memory224, and the register circuit228are all used to store data. In some embodiments, the external memory210may be a dynamic random access memory (DRAM), the memory224may be a static random access memory (SRAM), and the register circuit228may include multiple registers.

Reference is made toFIGS.3A to3C, which are flowcharts of the tensor concatenation method according to an embodiment of the present invention. The steps ofFIG.3Amay be performed by the DMA circuit223.FIG.3Arelates to reading data (i.e., a tensor or a part of a tensor) from the external memory210and writing the data to the memory224. The steps ofFIG.3Bmay be performed by the vector accelerator226.FIG.3Brelates to reading data from the memory224, writing data to the register circuit228, reading data from the register circuit228, and writing data to the memory224. The steps ofFIG.3Cmay be performed by the DMA circuit223.FIG.3Crelates to reading data from the memory224and writing the data to the external memory210.

FIGS.4A to4Dare schematic diagrams of tensor concatenation according to the present invention and correspond to the steps ofFIGS.3A to3C. Note that inFIGS.4A to4D, the amount of data in the memory224and the register circuit228is for illustrative purposes only. It does not mean that the memory224stores all of the data of each of the tensors to be concatenated at the same time, nor does it mean that the register circuit228stores all of the data of each of the tensors to be concatenated at the same time.

Reference is made to bothFIG.3AandFIG.4Afor the following discussion.FIG.3Aincludes the following steps.

Step S310: The DMA circuit223determines a target tensor among multiple tensors in the external memory210. As shown inFIG.4A, the external memory210stores4tensors to be concatenated (each is a3-dimensional tensor): a tensor211(of shape [100,32,4]), a tensor212(of shape [100,32,4]), a tensor213(of shape [100,32,1]), and a tensor214(of shape [100,32,1]). The first2dimensions of these4tensors ([100,32]) are the same, the third dimensions (i.e., the innermost dimension) are different. In this step, the DMA circuit223selects one of the tensor211, the tensor212, the tensor213, and the tensor214as a target tensor.

Step S320: The DMA circuit223reads a part of the target tensor from the external memory210. In this step, the DMA circuit223reads a first amount of data from consecutive addresses of the external memory210in the same one read operation. In some embodiments, the first amount of data may be the bandwidth of the external memory210(e.g., 32 bytes).

Step S330: The DMA circuit223stores the part of the target tensor into the memory224. In this step, the DMA circuit223writes a second amount of data into consecutive addresses of the memory224in the same one write operation. The second amount of data is less than or equal to the operating speed of the IPU220(for example, assuming that the IPU220processes 16, 32 or 64 bytes of data per unit time, the second amount of data is less than or equal to 16, 32 or 64 bytes). In some embodiments, the first amount of data is equal to the second amount of data; that is to say, the DMA circuit223writes the data read in the previous step into the memory224in one write operation.

Step S335: The DMA circuit223determines whether all tensors to be concatenated have been read. If YES, the flow ofFIG.3Aends; otherwise, steps S310to S330are repeated to move more tensors.

In some embodiments, because the memory224has limited space (i.e., its capacity is smaller than the capacity of the external memory210in order to reduce costs), the memory224cannot store all data of all of the tensors to be concatenated at the same time. However, the ratio of the amount of data of the tensors to be concatenated in the memory224is equal to the ratio of the innermost dimensions (i.e., the axis of the concatenation operation) of the tensors to be concatenated. For example, because the ratio of the innermost dimensions of the tensor211, the tensor212, the tensor213, and the tensor214is 4:4:1:1, the ratio of the amount of data of the tensor211, the tensor212, the tensor213, and the tensor214in the memory224is also 4:4:1:1. In other words, in some embodiments, the DMA circuit223determines the target tensor in step S310according to the ratio of the dimensions corresponding to the axis of the concatenation operation. For example, the tensor212is selected each time the tensor211is selected, and the tensor214is selected each time the tensor213is selected, but the tensor213(or the tensor214) is selected only once every4times the tensor211(or the tensor212) is selected.

Reference is made toFIG.3B,FIG.4B, andFIG.4Cfor the following discussion.FIG.3Bincludes the following steps.

Step S340: The vector accelerator226determines a target tensor among multiple tensors in the memory224. TakingFIG.4Bas an example, the vector accelerator226selects one of the tensor211, the tensor212, the tensor213, and the tensor214as the target tensor.

Step S350: The vector accelerator226stores a part of the target tensor in at least one row of the register circuit228. Step S350includes sub-step S352and sub-step S354.

Step S352: The vector accelerator226reads N bytes of the target tensor (N is the aforementioned second amount of data). More specifically, the vector accelerator226may read N bytes in the same one read operation, and the N bytes may be stored in consecutive addresses of the memory224.

Step S354: The vector accelerator226writes the N bytes into M rows of the register circuit228. TakingFIG.4Bas an example, for the tensor211, the N bytes are written into columns1to4of the register circuit228, occupying a total of M=N/4 rows (4is the innermost dimension of the tensor211). More specifically, the 1st to 4thbytes of the N bytes are respectively written into columns1to4of the row R11of the register circuit228, and the5th to8th bytes of the N bytes are respectively written into columns1to4of the row R12(which is next to the row R11) of the register circuit228, . . . , and so on. Similarly, for the tensor212, the N bytes are written into columns5to8of the register circuit228, occupying a total of M=N/4 rows (4 is the innermost dimension of the tensor212); for the tensor213, the N bytes are written into column9of the register circuit228, occupying a total of M=N/1 rows (1 is the innermost dimension of the tensor213); for the tensor214, the N bytes are written into column10of the register circuit228, occupying a total of M=N/1 rows (1 is the innermost dimension of the tensor214). In other words, M is equal to N divided by the innermost dimension of the target tensor. In some embodiments, N is a common multiple of the innermost dimensions of all of the tensors to be concatenated.

Step S360: The vector accelerator226determines whether a target row of the register circuit228contains partial data of each of the tensors to be concatenated. If YES, the vector accelerator226performs step S370; otherwise, the vector accelerator226performs step S340. TakingFIG.4Bas an example, assuming that the target row is the first row (R11) of the register circuit228, after the vector accelerator226has performed step S340and step S350once on each of the tensor211, the tensor212, the tensor213, and the tensor214, the target row contains partial data of each of the tensors to be concatenated.

Step S370: The vector accelerator226writes the data of the target row to the memory224. Reference is made toFIG.4C. In this step, the vector accelerator226writes the target row (e.g., the row R11, the row R21, the row R31, or the row R41, which may be the first row of the data group GP1, the data group GP2, the data group GP3, and the data group GP4respectively) to the memory224. In some embodiments, when the amount of effective data DV of the target row is less than or equal to 1/L times the maximum amount of data (e.g., BW bytes, which may be equal to the aforementioned second amount of data) by which the vector accelerator226performs a read operation or write operation on the memory224, the vector accelerator226can use at most L ports simultaneously to move data to save time. The amount of effective data DV of the target row is the number of bytes of the effective data in a row and is equal to the sum of the innermost dimensions of all tensors to be concatenated, which is the innermost dimension of the concatenated tensor. TakingFIG.4Cas an example (assuming BW=32), because L=[BW/DV]=[32/10]=3, the vector accelerator226can use at most3ports at the same time to move data (inFIG.4C,2ports are illustrated as an example; however, it is also possible to use3ports or only1port). More specifically, the vector accelerator226moves the data of the odd-numbered data groups (the data group GP1, the data group GP3, . . . ) in the register circuit228to the data block DB1in the memory224through the port PT1, and move the data of the even-numbered data groups (the data group GP2, the data group GP4, . . . ) in the register circuit228to the data block DB2in the memory224through the port PT2. The amount of data of each data group can be the same or different.

In some embodiments, the vector accelerator226writes the row R11and the row R21to the memory224at substantially the same time.

In some embodiments, step S370may be performed simultaneously with steps S340to S350. That is to say, the vector accelerator226can move the tensors to be concatenated in the memory224to the register circuit228(FIG.4B) while moving the concatenated intermediate data (i.e., a part of the concatenated tensor, such as, a data group or a row of a data group) to the memory224(FIG.4C).

Reference is made to bothFIG.3CandFIG.4Dfor the following discussion.FIG.3Cincludes the following steps.

Step S380: The DMA circuit223reads the effective data in a row of the memory224. For example, in this step, the DMA circuit223reads the effective data E11, the effective data E21, or the effective data E31(which are respectively the amount of effective data DV in the row R11, the row R21, and the row R31).

Step S390: The DMA circuit223stores the effective data in the external memory210. For example, the DMA circuit223writes the effective data E11, the effective data E21, or the effective data E31in the memory224to the corresponding location (or address) in the external memory210to become a part of the concatenated tensor215.

Note that the memory224also actually stores a part of the concatenated tensor, with the data arrangement in the memory224being different from the data arrangement in the external memory210. In other words, in some embodiments, the convolution engine221and/or the vector engine222of the IPU220can directly read the concatenated data in the memory224for subsequent operations.

Step S395: The DMA circuit223determines whether all effective data in the memory224has been moved to the external memory210. If YES, the process ofFIG.3Cends; otherwise, the flow returns to step S380.

To sum up, the present invention greatly speeds up tensor concatenation. For example, the conventional method requires a total of 100*32*32=102400 operation cycles to concatenate 32 tensors of the shape [100,32,1] into one tensor of the shape [100,32,32]. In comparison, to concatenate the same tensors, the method of the present invention requires a total of 100*32/32*32+100* (32+16) +100*32=3200+4800+3200 =11200 operation cycles (3200,4800, and3200correspond toFIG.3A,FIG.3B, andFIG.3Crespectively). The time required by the conventional method is 102400/11200≈9.14 times that of the present invention.

Reference is made toFIG.5, which is a schematic diagram of using a multi-stage pipeline according to an embodiment of the present invention. In this embodiment, the DMA circuit223includes a channel510and a channel520, which are used to perform the process ofFIG.3Aand the process ofFIG.3Crespectively. In this way, as shown inFIG.5, the process ofFIG.3A, the process ofFIG.3B, and the process ofFIG.3Ccan be performed at substantially the same time, improving the tensor concatenation speed of the present invention. Continuing the above example (where32tensors of the shape [100,32,1] are to be concatenated), the multi-stage pipeline inFIG.5requires only about4800operation cycles (corresponding to the total time consumption of the process inFIG.3B).

FIG.6shows a schematic diagram of the register circuit228according to an embodiment of the present invention. The register circuit228is a register array including a plurality of registers REG (e.g., each register REG is one bit), and each register REG has its own write line and read line. In this way, the vector accelerator226can access any number and any position of the register(s) REG in the register circuit228in each operation cycle, which greatly improves the flexibility of read operations and write operations. In comparison, because a row of memory cells in an SRAM shares one write line and one read line, the read and write operations of the SRAM have greater limitations.

The number of tensors to be concatenated (which is4in the discussions above) is intended to illustrate the invention by way of example and not to limit the scope of the claimed invention. People having ordinary skill in the art may apply the present invention to2,3, or more tensors in accordance with the foregoing discussions.

The axis of the concatenation operation being the innermost dimension of the tensors is intended to illustrate the invention by way of example and not to limit the scope of the claimed invention. People having ordinary skill in the art may apply the present invention to a case where the axis of the concatenation operation is not the innermost dimension of the tensor in accordance with the foregoing discussions.