Patent ID: 11941514
Assignee: ZHEJIANG LAB
Field: Computer technology (Electrical engineering)
Classification: CPC G  Y | IPC G

Claim 10:
11. An apparatus for execution of a computational graph in a neural network model, comprising the following modules:
a task execution body construction module, configured for creating a task queue of operator kernel functions, threads of task execution bodies, task execution bodies of kernel functions, an event recall queue, and a thread of the event recall queue;
a task execution body internal module, configured for: by a current execution body, producing tensor data of the current execution body, packing the tensor data into a message, and sending the message to an upstream execution body and a downstream execution body; by the downstream execution body, preparing tensor data, preparing an available tensor memory block required for storing data produced by the downstream execution body, and executing a task of an internal operator kernel function of the downstream execution body and producing output tensor data; and
a task execution body pipelining and parallelizing module, configured for allocating idle memory blocks to the respective task execution bodies, initiating execution of tasks, making execution bodies perform the entire computational graph in parallel, wherein a specific process executed by the task execution body pipelining and parallelizing module is as follows:
step 1: allocating idle memory blocks to execution bodies, wherein a physical computational graph composed of a plurality of operators having production and consumption relationships is constructed, the operators is labeled as operator a, operator b, operator c, . . . , and operator i; execution bodies for executing their own kernel functions are created according to respective operators to constitute an execution computational graph composed of execution body A, execution body B, execution body C, . . . , and execution body I having production and consumption relationships; and an idle memory block for tensors produced and consumed by each execution body is allocated to the corresponding execution body;
feeding different batches of input data, wherein a memory is allocated for tensor data produced when execution body A executes a kernel function of operator a, wherein the idle memory block corresponding to a zeroth batch of data is memory block a0, the idle memory block corresponding to a first batch of data is memory block a1, the idle memory block corresponding to a second batch of data is memory block a2, . . . , and the idle memory block corresponding to an ith batch of data is memory ai;
allocating a memory for tensor data produced when execution body B executes a kernel function of operator b, wherein the idle memory block corresponding to the zeroth batch of data is memory block b0, the idle memory block corresponding to the first batch of data is memory block b1, the idle memory block corresponding to the second batch of data is memory block b2, . . . , and the idle memory block corresponding to the ith batch of data is memory block bi;
allocating a memory for tensor data produced when execution body C executes a kernel function of operator c, wherein the idle memory block corresponding to the zeroth batch of data is memory block c0, the idle memory block corresponding to the first batch of data is memory block c1, the idle memory block corresponding to the second batch of data is memory block c2, . . . , and the idle memory block corresponding to the ith batch of data is memory block ci;
repeating the memory allocation process until a memory block is allocated for tensor data produced when execution body I executes a kernel function of operator i;
step 2: initiating execution of an execution body, wherein at time T0, the zeroth batch of data is input, execution body A executes the kernel function of operator a and writes the output tensor data as an execution result into idle memory block a0, and downstream execution body B, execution body C, . . . , and execution body I are in a waiting state since there are no readable input tensor data;
step 3: performing the entire computational graph in parallel, wherein at time T1, execution body A informs execution body B of reading memory block a0 produced by execution body A; execution body B receives a message of reading memory block a0 produced by execution body A, and checks whether there is an idle memory block available in memory region b produced by execution body B; if it finds that there is idle memory block b0 available, execution body B executes the computational task of the kernel function of operator b and reads memory block a0; execution body B writes an output tensor result generated by execution into memory block b0; at time T1, execution body A also checks whether execution body A has a writable idle memory block; if execution body A has the writable idle memory block, at time T1, execution body A also executes the first batch of input data and writes an execution result into idle memory block a1, so that execution body A and execution body B start to operate in parallel; and downstream execution body C, . . . , and execution body I are still in the waiting state since there are no readable data;
at time T2, execution body B sends a message to a downstream consumer, i.e. execution body C, to inform execution body C of reading memory block b0 after execution body B produces memory block b0; at time T2, execution body B sends a message to an upstream producer, i.e. execution body A, to inform execution body A of a fact that execution body B has used memory block a0 of execution body A; at time T2, execution body A sends memory block a1 produced for training the first batch of input data to execution body B for a second time; execution body B checks to find that it still has idle memory block b1, thus starts to read memory block a1 and write into idle memory block b1; execution body C receives memory block b0 and finds that it has an idle memory block c0 available, then starts to read memory block b0 and write into memory block c0; execution body A receives a message of memory block a0 having been used and returned by execution body B and checks to find that all of consumers of execution body A have used memory block a0, then execution body A recovers memory block a0 and labels memory block a0 as an idle block; execution body A also continues to execute and write into memory block a2;
at time T2, execution bodies A, B, and C all operate, wherein for a deep learning training task, at time T2, memory block b0 and memory block c0 store the zeroth batch of data for training; memory block a1 and memory block b1 store the first batch of data for training; memory block a2 stores the second batch of data for training; and
all the execution bodies are operated in a pipeline parallel manner by step 3.